[jira] [Commented] (OMID-103) DeleteFamily marker still visible after rollback fails
[ https://issues.apache.org/jira/browse/OMID-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537717#comment-16537717 ] James Taylor commented on OMID-103: --- FYI, [~ohads] - the specific scenario in this case is: * HBase write of family delete marker occurs successfully * Commit was successful * Write of shadow cells failed * Query/scan over rows doesn't properly filter out all deleted cells for the family delete marker > DeleteFamily marker still visible after rollback fails > -- > > Key: OMID-103 > URL: https://issues.apache.org/jira/browse/OMID-103 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > > If a DeleteFamily marker is written to HBase and the subsequent abort of the > transaction fails, the delete marker is still visible and mistakenly > continues to take effect. For a repro, see > TransactionIT.testFailureToRollbackAfterDelete() which passes for Tephra but > fails for Omid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537305#comment-16537305 ] Lianghong Xu commented on OMID-90: -- Hi [~ohads] [~ebortnik] thanks for providing the low latency implementation and sorry for the late response. We have not yet got a chance to test it due to lack of resources since it's near end of H1. We will try to do the benchmark by the end of July and will keep you updated. Thanks! > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536912#comment-16536912 ] Edward Bortnikov commented on OMID-90: -- [~lianghongxu], have you had a chance to play with the new protocol? Any feedback that you can share? Your experience matters as we move on to make the low-latency protocol part of Omid's next release. > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (OMID-102) Implement visibility filter as pure HBase Filter
[ https://issues.apache.org/jira/browse/OMID-102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ohad Shacham reassigned OMID-102: - Assignee: Yonatan Gottesman > Implement visibility filter as pure HBase Filter > > > Key: OMID-102 > URL: https://issues.apache.org/jira/browse/OMID-102 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Major > > The way Omid currently filters through it's own RegionScanner won't work the > way it's implemented (i.e. the way the filtering is done *after* the next > call). The reason is that the state of HBase filters get messed up since > these filters will start to see cells that it shouldn't (i.e. cells that > would be filtered based on snapshot isolation). It cannot be worked around by > manually running filters afterwards because filters may issue seek calls > which are handled during the running of scans by HBase. > > Instead, the filtering needs to be implemented as a pure HBase filter and > that filter needs to delegate to the other, delegate filter once it's > determined that the cell is visible. See Tephra's TransactionVisibilityFilter > and they way it calls the delegate filter (cellFilters) only after it's > determined that the cell is visible. You may run into TEPHRA-169 without > including the CellSkipFilter too. > Because it'll be easier if you see shadow cells *before* their corresponding > real cells you can prefix instead of suffix the column qualifiers to > guarantee that you'd see the shadow cells prior to the actual cells. Or you > could buffer cells in your filter prior to omitting them. Another issue would > be if the shadow cells aren't found and you need to consult the commit table > - I suppose if the shadow cells are first, this logic would be easier to know > when it needs to be called. > > To reproduce, see the Phoenix unit tests > FlappingTransactionIT.testInflightUpdateNotSeen() and > testInflightDeleteNotSeen(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-102) Implement visibility filter as pure HBase Filter
[ https://issues.apache.org/jira/browse/OMID-102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ohad Shacham updated OMID-102: -- Issue Type: Sub-task (was: Bug) Parent: OMID-82 > Implement visibility filter as pure HBase Filter > > > Key: OMID-102 > URL: https://issues.apache.org/jira/browse/OMID-102 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > > The way Omid currently filters through it's own RegionScanner won't work the > way it's implemented (i.e. the way the filtering is done *after* the next > call). The reason is that the state of HBase filters get messed up since > these filters will start to see cells that it shouldn't (i.e. cells that > would be filtered based on snapshot isolation). It cannot be worked around by > manually running filters afterwards because filters may issue seek calls > which are handled during the running of scans by HBase. > > Instead, the filtering needs to be implemented as a pure HBase filter and > that filter needs to delegate to the other, delegate filter once it's > determined that the cell is visible. See Tephra's TransactionVisibilityFilter > and they way it calls the delegate filter (cellFilters) only after it's > determined that the cell is visible. You may run into TEPHRA-169 without > including the CellSkipFilter too. > Because it'll be easier if you see shadow cells *before* their corresponding > real cells you can prefix instead of suffix the column qualifiers to > guarantee that you'd see the shadow cells prior to the actual cells. Or you > could buffer cells in your filter prior to omitting them. Another issue would > be if the shadow cells aren't found and you need to consult the commit table > - I suppose if the shadow cells are first, this logic would be easier to know > when it needs to be called. > > To reproduce, see the Phoenix unit tests > FlappingTransactionIT.testInflightUpdateNotSeen() and > testInflightDeleteNotSeen(). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
