[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-4344: --- Resolution: Fixed Status: Resolved (was: Patch Available) included in 2856-v6.txt Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v12.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-4344: --- Attachment: 4344-v12.txt combine 4344-v11.txt with 4485-v4.txt Running the testsuite (in progress) Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v12.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v9.txt Patch v8 contains a slightly different version of patch for HBASE-4345 Patch v9 pulls in the changes from HBASE-4345. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-4344: --- Attachment: 4344-v10.txt explictly add KEY_VALUE_VERSION to the file info to keep track of weather we are using the new format in the HFile or not. We can bump this up, if we later want to change the format in which the KV is stored. Say to accomidate a variable length encoding, or other optimisation/approaches. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v10.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v11.txt It enables the tests from TestAcidGuarantees which pass. I also introduced a constant, KEY_VALUE_VER_WITH_MEMSTORE, for my earlier comment. Will run through test suite once TRUNK build stabilizes. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: (was: 4344-v11.txt) Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v11.txt Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v10.txt, 4344-v11.txt, 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, 4344-v8.txt, 4344-v9.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v5.txt Patch v5. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v6.txt Patch v6 fixes ArrayIndexOutOfBoundsException mentioned above. Still need to investigate 'Already used this rwcc. Too late to initialize' issue. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: (was: 4344-v6.txt) Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v6.txt Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v7.txt Patch version 7 passes all unit tests. I used Lars' patch for TestStoreFile. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, 4344-v4.txt, 4344-v5.txt, 4344-v6.txt, 4344-v7.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[
https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-4344:
--
Comment: was deleted
(was: boolean includesMemTS parameter was only added to selected KeyValue ctors.
Some ctors weren't covered:
e.g.
{code}
public KeyValue(final byte [] row, final byte [] family,
final byte [] qualifier, final byte [] value) {
{code})
Persist memstoreTS to disk
--
Key: HBASE-4344
URL: https://issues.apache.org/jira/browse/HBASE-4344
Project: HBase
Issue Type: Sub-task
Reporter: Amitanand Aiyer
Assignee: Amitanand Aiyer
Fix For: 0.89.20100924
Attachments: 4344-v2.txt, patch-2
Atomicity can be achieved in two ways -- (i) by using a multiversion
concurrency system (MVCC), or (ii) by ensuring that new writes do not
complete, until the old reads complete.
Currently, Memstore uses something along the lines of MVCC (called RWCC for
read-write-consistency-control). But, this mechanism is not incorporated for
the key-values written to the disk, as they do not include the memstore TS.
Let us make the two approaches be similar, by persisting the memstoreTS along
with the key-value when it is written to the disk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-4344: - Attachment: 4344-v4.txt He Ted's patch with all the nasty casting to HFileReaderV2 removed. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, 4344-v4.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-4344: -- Attachment: 4344-v2.txt Patch version 2 is rebased for TRUNK. Running test suite now. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: 4344-v2.txt, patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-4344: --- Fix Version/s: (was: 0.94.0) 0.89.20100924 Assignee: Amitanand Aiyer Status: Patch Available (was: Open) Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4344) Persist memstoreTS to disk
[ https://issues.apache.org/jira/browse/HBASE-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amitanand Aiyer updated HBASE-4344: --- Attachment: patch-2 There are three parts to the changes made here: The first one is in the write path, while writing the HFile (V2). We include the memstoreTS along with the other information already written during the KeyValue. We maintain and append to the file info the maximum memstoreTS found among all the key-values written to the disk. The presence of this information in the file info represents the fact that the Key-values written to the HFile is in the new format that includes the memstoreTS. Part 2: The reader is modified accordingly to see if the file info suggests the presence of the memstoreTS or not. If it does, then we also read the memstoreTS while preparing the KeyValues in the reader. Part 3: On Region open/initialize, we need to ensure that the RWCC is initialized with a value bigger than the max of any value found in the storeFiles. this is necessary to ensure that the writes in the storefiles are seen by future reads. Persist memstoreTS to disk -- Key: HBASE-4344 URL: https://issues.apache.org/jira/browse/HBASE-4344 Project: HBase Issue Type: Sub-task Reporter: Amitanand Aiyer Assignee: Amitanand Aiyer Fix For: 0.89.20100924 Attachments: patch-2 Atomicity can be achieved in two ways -- (i) by using a multiversion concurrency system (MVCC), or (ii) by ensuring that new writes do not complete, until the old reads complete. Currently, Memstore uses something along the lines of MVCC (called RWCC for read-write-consistency-control). But, this mechanism is not incorporated for the key-values written to the disk, as they do not include the memstore TS. Let us make the two approaches be similar, by persisting the memstoreTS along with the key-value when it is written to the disk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
