[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r224179421
--- Diff:
hbase-client/src/test/java/org/apache/omid/transaction/TestOmidLLRaces.java ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+
+import static com.google.common.base.Charsets.UTF_8;
+import static
org.apache.omid.committable.hbase.HBaseCommitTableConfig.DEFAULT_COMMIT_TABLE_CF_NAME;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+
+import com.google.common.base.Optional;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.omid.committable.CommitTable;
+import org.apache.omid.committable.hbase.KeyGenerator;
+import org.apache.omid.committable.hbase.KeyGeneratorImplementations;
+import org.apache.omid.metrics.NullMetricsProvider;
+import org.apache.omid.tso.client.OmidClientConfiguration;
+import org.apache.omid.tso.client.TSOClient;
+import org.apache.omid.tso.client.TSOFuture;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+import org.testng.ITestContext;
+import org.testng.annotations.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+
+@Test(groups = "sharedHBase")
+public class TestOmidLLRaces extends OmidTestBase {
+
+private static final byte[] row1 = Bytes.toBytes("test-is-committed1");
+private static final byte[] row2 = Bytes.toBytes("test-is-committed2");
+private static final byte[] family = Bytes.toBytes(TEST_FAMILY);
+private static final byte[] qualifier = Bytes.toBytes("testdata");
+private static final byte[] data1 = Bytes.toBytes("testWrite-1");
+
+private static final Logger LOG =
LoggerFactory.getLogger(TestOmidLLRaces.class);
+@Override
+protected boolean isLowLatency() {
+return true;
+}
+
+@Test(timeOut = 30_000)
+public void testIsCommitted(ITestContext context) throws Exception {
+AbstractTransactionManager tm =
(AbstractTransactionManager)newTransactionManagerHBaseCommitTable(getClient(context));
+
+Table htable = connection.getTable(TableName.valueOf(TEST_TABLE));
+SnapshotFilterImpl snapshotFilter = new SnapshotFilterImpl(new
HTableAccessWrapper(htable, htable),
+tm.getCommitTableClient());
+TTable table = spy(new TTable(htable, snapshotFilter, false));
+
+HBaseTransaction t1 = (HBaseTransaction) tm.begin();
+
+Put put = new Put(row1);
+put.addColumn(family, qualifier, data1);
+table.put(t1, put);
+tm.commit(t1);
+
+HBaseTransaction t2 = (HBaseTransaction) tm.begin();
+put = new Put(row2);
+put.addColumn(family, qualifier, data1);
+table.put(t2, put);
+table.flushCommits();
+
+HBaseTransaction t3 = (HBaseTransaction) tm.begin();
+put = new Put(row2);
+put.addColumn(family, qualifier, data1);
+table.put(t3, put);
+tm.commit(t3);
+
+HBaseCellId hBaseCellId1 = new HBaseCellId(table, row1, family,
qualifier, t1.getStartTimestamp());
+HBaseCellId hBaseCellId2 = new HBaseCellId(table, row2, family,
qualifier, t2.getStartTimestamp());
+HBaseCellId hBaseCellId3 = new HBaseCellId(table, row2, family,
qualifier,
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645330#comment-16645330
]
ASF GitHub Bot commented on OMID-90:
Github user JamesRTaylor commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r224179421
--- Diff:
hbase-client/src/test/java/org/apache/omid/transaction/TestOmidLLRaces.java ---
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+
+import static com.google.common.base.Charsets.UTF_8;
+import static
org.apache.omid.committable.hbase.HBaseCommitTableConfig.DEFAULT_COMMIT_TABLE_CF_NAME;
+import static org.mockito.Mockito.doAnswer;
+import static org.mockito.Mockito.spy;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+import static org.mockito.Matchers.any;
+
+import com.google.common.base.Optional;
+import org.apache.hadoop.hbase.TableName;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.omid.committable.CommitTable;
+import org.apache.omid.committable.hbase.KeyGenerator;
+import org.apache.omid.committable.hbase.KeyGeneratorImplementations;
+import org.apache.omid.metrics.NullMetricsProvider;
+import org.apache.omid.tso.client.OmidClientConfiguration;
+import org.apache.omid.tso.client.TSOClient;
+import org.apache.omid.tso.client.TSOFuture;
+import org.mockito.invocation.InvocationOnMock;
+import org.mockito.stubbing.Answer;
+import org.testng.ITestContext;
+import org.testng.annotations.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Set;
+import java.util.concurrent.CountDownLatch;
+
+
+@Test(groups = "sharedHBase")
+public class TestOmidLLRaces extends OmidTestBase {
+
+private static final byte[] row1 = Bytes.toBytes("test-is-committed1");
+private static final byte[] row2 = Bytes.toBytes("test-is-committed2");
+private static final byte[] family = Bytes.toBytes(TEST_FAMILY);
+private static final byte[] qualifier = Bytes.toBytes("testdata");
+private static final byte[] data1 = Bytes.toBytes("testWrite-1");
+
+private static final Logger LOG =
LoggerFactory.getLogger(TestOmidLLRaces.class);
+@Override
+protected boolean isLowLatency() {
+return true;
+}
+
+@Test(timeOut = 30_000)
+public void testIsCommitted(ITestContext context) throws Exception {
+AbstractTransactionManager tm =
(AbstractTransactionManager)newTransactionManagerHBaseCommitTable(getClient(context));
+
+Table htable = connection.getTable(TableName.valueOf(TEST_TABLE));
+SnapshotFilterImpl snapshotFilter = new SnapshotFilterImpl(new
HTableAccessWrapper(htable, htable),
+tm.getCommitTableClient());
+TTable table = spy(new TTable(htable, snapshotFilter, false));
+
+HBaseTransaction t1 = (HBaseTransaction) tm.begin();
+
+Put put = new Put(row1);
+put.addColumn(family, qualifier, data1);
+table.put(t1, put);
+tm.commit(t1);
+
+HBaseTransaction t2 = (HBaseTransaction) tm.begin();
+put = new Put(row2);
+put.addColumn(family, qualifier, data1);
+table.put(t2, put);
+table.flushCommits();
+
+HBaseTransaction t3 = (HBaseTransaction) tm.begin();
+put = new Put(row2);
+put.addColumn(family, qualifier, data1);
+table.put(t3, put);
+tm.commit(t3);
+
+HBaseCellId hBaseCellId1 = new HBaseCellId(table, row1, family,
qualifier,
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223974301
--- Diff:
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long
cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether
the transaction was invalidated
-Optional commitTimeStamp =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
-commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+// In case of LL, if found invalid ct cell, still must check
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
// 4) Check the epoch and invalidate the entry
--- End diff --
or invalidate in a non latency mode
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223993765
--- Diff:
tso-server/src/main/java/org/apache/omid/tso/RequestProcessorSkipCT.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.tso;
+
+import com.google.inject.Inject;
+import org.apache.omid.metrics.MetricsRegistry;
+import org.jboss.netty.channel.Channel;
+
+import java.io.IOException;
+
+public class RequestProcessorSkipCT extends AbstractRequestProcessor {
+
+
+private final ReplyProcessor replyProcessor;
+
+private final LeaseManagement leaseManager;
+private final Panicker panicker;
+private final String tsoHostAndPort;
+
+@Inject
+RequestProcessorSkipCT(MetricsRegistry metrics,
+ TimestampOracle timestampOracle,
+ ReplyProcessor replyProcessor,
+ Panicker panicker,
+ LeaseManagement leaseManager,
+ TSOServerConfig config,
+ LowWatermarkWriter lowWatermarkWriter,
+ String tsoHostAndPort) throws IOException {
+super(metrics, timestampOracle, panicker, config,
lowWatermarkWriter);
+this.replyProcessor = replyProcessor;
+this.tsoHostAndPort = tsoHostAndPort;
+requestRing = disruptor.start();
+this.leaseManager = leaseManager;
+this.panicker = panicker;
+}
+
+private void commitSuicideIfNotMaster() {
+if (!leaseManager.stillInLeasePeriod()) {
+panicker.panic("Replica " + tsoHostAndPort + " lost mastership
whilst flushing data. Committing suicide");
+}
+}
+
+@Override
+public void forwardCommit(long startTimestamp, long commitTimestamp,
Channel c, MonitoringContext monCtx) {
+commitSuicideIfNotMaster();
--- End diff --
Add a comment that his is required since returning commit results when the
TSO is not the leader might violate snapshot isolation. This is because we have
to guarantee that committing transaction from previous tso has to persist all
its data because a new transaction started by a new tso.
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223989230
--- Diff:
transaction-client/src/main/java/org/apache/omid/transaction/AbstractTransactionManager.java
---
@@ -350,6 +356,43 @@ private void
markReadOnlyTransaction(AbstractTransaction readO
}
+private void commitLowLatencyTransaction(AbstractTransaction tx)
+throws RollbackException, TransactionException {
+try {
+
+long commitTs = tsoClient.commit(tx.getStartTimestamp(),
tx.getWriteSet(), tx.getConflictFreeWriteSet()).get();
+boolean committed =
commitTableWriter.atomicAddCommittedTransaction(tx.getStartTimestamp(),commitTs);
+if (!committed) {
+// Transaction has been invalidated by other client
+rollback(tx);
+
commitTableClient.completeTransaction(tx.getStartTimestamp());
+rolledbackTxsCounter.inc();
+throw new RollbackException("Transaction " +
tx.getTransactionId() + " got invalidated");
+}
+certifyCommitForTx(tx, commitTs);
+updateShadowCellsAndRemoveCommitTableEntry(tx, postCommitter);
+
+} catch (ExecutionException e) {
+if (e.getCause() instanceof AbortException) { // TSO reports
Tx conflicts as AbortExceptions in the future
+rollback(tx);
+rolledbackTxsCounter.inc();
+throw new RollbackException("Conflicts detected in tx
writeset", e.getCause());
+}
+
+if (e.getCause() instanceof ServiceUnavailableException ||
e.getCause() instanceof ConnectionException) {
+errorTxsCounter.inc();
+rollback(tx); // Rollback proactively cause it's likely
that a new TSOServer is now master
--- End diff --
the leader
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223975170
--- Diff:
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long
cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether
the transaction was invalidated
-Optional commitTimeStamp =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
-commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+// In case of LL, if found invalid ct cell, still must check
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
// 4) Check the epoch and invalidate the entry
// if the data was written by a transaction from a previous
epoch (previous TSO)
-if (cellStartTimestamp < epoch) {
+if (cellStartTimestamp < epoch || isLowLatency) {
boolean invalidated =
commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
if (invalidated) { // Invalid commit timestamp
+
+// If we are running lowLatency Omid, we could have
manged to invalidate a ct entry,
+// but the committing client already wrote to shadow
cells:
--- End diff --
This can happen only in low latency mode, since in the regular mode the
client keeps the commit table entry if persisting the commit was done after the
tso lost its lease.
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223974621
--- Diff:
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long
cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether
the transaction was invalidated
-Optional commitTimeStamp =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
-commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+// In case of LL, if found invalid ct cell, still must check
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
// 4) Check the epoch and invalidate the entry
// if the data was written by a transaction from a previous
epoch (previous TSO)
-if (cellStartTimestamp < epoch) {
+if (cellStartTimestamp < epoch || isLowLatency) {
boolean invalidated =
commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
if (invalidated) { // Invalid commit timestamp
+
+// If we are running lowLatency Omid, we could have
manged to invalidate a ct entry,
--- End diff --
managed
---
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223971768 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e // 2) Then check the commit table // If the data was written at a previous epoch, check whether the transaction was invalidated --- End diff -- This "If the data was written at a previous epoch," can be removed. ---
[jira] [Updated] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonatan Gottesman updated OMID-109: --- Attachment: omid109_v2.patch > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644682#comment-16644682
]
ASF GitHub Bot commented on OMID-90:
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223989230
--- Diff:
transaction-client/src/main/java/org/apache/omid/transaction/AbstractTransactionManager.java
---
@@ -350,6 +356,43 @@ private void
markReadOnlyTransaction(AbstractTransaction readO
}
+private void commitLowLatencyTransaction(AbstractTransaction tx)
+throws RollbackException, TransactionException {
+try {
+
+long commitTs = tsoClient.commit(tx.getStartTimestamp(),
tx.getWriteSet(), tx.getConflictFreeWriteSet()).get();
+boolean committed =
commitTableWriter.atomicAddCommittedTransaction(tx.getStartTimestamp(),commitTs);
+if (!committed) {
+// Transaction has been invalidated by other client
+rollback(tx);
+
commitTableClient.completeTransaction(tx.getStartTimestamp());
+rolledbackTxsCounter.inc();
+throw new RollbackException("Transaction " +
tx.getTransactionId() + " got invalidated");
+}
+certifyCommitForTx(tx, commitTs);
+updateShadowCellsAndRemoveCommitTableEntry(tx, postCommitter);
+
+} catch (ExecutionException e) {
+if (e.getCause() instanceof AbortException) { // TSO reports
Tx conflicts as AbortExceptions in the future
+rollback(tx);
+rolledbackTxsCounter.inc();
+throw new RollbackException("Conflicts detected in tx
writeset", e.getCause());
+}
+
+if (e.getCause() instanceof ServiceUnavailableException ||
e.getCause() instanceof ConnectionException) {
+errorTxsCounter.inc();
+rollback(tx); // Rollback proactively cause it's likely
that a new TSOServer is now master
--- End diff --
the leader
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644680#comment-16644680
]
ASF GitHub Bot commented on OMID-90:
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223976065
--- Diff:
hbase-client/src/test/java/org/apache/omid/transaction/TestEndToEndScenariosWithHA.java
---
@@ -287,7 +287,7 @@ public void testScenario1() throws Exception {
// TX 2 modifies cells R1C1 & R2C2 (v2)
// TX 2 commits
// End of Test state: R1C1 & R2C2 (v2)
-@Test(timeOut = 60_000)
+@Test(timeOut = 60_00)
--- End diff --
Why this is needed? Due to the low_cpu mode? or you just forgot it? :)
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644676#comment-16644676
]
ASF GitHub Bot commented on OMID-90:
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223975170
--- Diff:
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long
cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether
the transaction was invalidated
-Optional commitTimeStamp =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
-commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+// In case of LL, if found invalid ct cell, still must check
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
// 4) Check the epoch and invalidate the entry
// if the data was written by a transaction from a previous
epoch (previous TSO)
-if (cellStartTimestamp < epoch) {
+if (cellStartTimestamp < epoch || isLowLatency) {
boolean invalidated =
commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
if (invalidated) { // Invalid commit timestamp
+
+// If we are running lowLatency Omid, we could have
manged to invalidate a ct entry,
+// but the committing client already wrote to shadow
cells:
--- End diff --
This can happen only in low latency mode, since in the regular mode the
client keeps the commit table entry if persisting the commit was done after the
tso lost its lease.
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644678#comment-16644678
]
ASF GitHub Bot commented on OMID-90:
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223974621
--- Diff:
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long
cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether
the transaction was invalidated
-Optional commitTimeStamp =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
-commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+// In case of LL, if found invalid ct cell, still must check
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
// 4) Check the epoch and invalidate the entry
// if the data was written by a transaction from a previous
epoch (previous TSO)
-if (cellStartTimestamp < epoch) {
+if (cellStartTimestamp < epoch || isLowLatency) {
boolean invalidated =
commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
if (invalidated) { // Invalid commit timestamp
+
+// If we are running lowLatency Omid, we could have
manged to invalidate a ct entry,
--- End diff --
managed
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644679#comment-16644679
]
ASF GitHub Bot commented on OMID-90:
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223973121
--- Diff:
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long
cellStartTimestamp, long e
// 2) Then check the commit table
// If the data was written at a previous epoch, check whether
the transaction was invalidated
-Optional commitTimeStamp =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT =
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
}
// 3) Read from shadow cell
-commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp =
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
if (commitTimeStamp.isPresent()) {
return commitTimeStamp.get();
}
+// In case of LL, if found invalid ct cell, still must check
sc in stage 3 then return
--- End diff --
Please add this comment: "This check is not needed in a non low latency
mode, since during post commit, the client keeps the entry in the commit table
when persisting the commit was done after the TSO lost its lease."
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644677#comment-16644677 ] ASF GitHub Bot commented on OMID-90: Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223971768 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e // 2) Then check the commit table // If the data was written at a previous epoch, check whether the transaction was invalidated --- End diff -- This "If the data was written at a previous epoch," can be removed. > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644681#comment-16644681 ] ASF GitHub Bot commented on OMID-90: Github user ohadshacham commented on a diff in the pull request: https://github.com/apache/incubator-omid/pull/46#discussion_r223975342 --- Diff: hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java --- @@ -240,7 +261,8 @@ public CommitTimestamp locateCellCommitTimestamp(long cellStartTimestamp, long e CellUtil.cloneQualifier(cell), cell.getTimestamp()), commitCache, -tableAccessWrapper)); +tableAccessWrapper), +isLowLatency); --- End diff -- indent > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644684#comment-16644684
]
ASF GitHub Bot commented on OMID-90:
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r223993765
--- Diff:
tso-server/src/main/java/org/apache/omid/tso/RequestProcessorSkipCT.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.tso;
+
+import com.google.inject.Inject;
+import org.apache.omid.metrics.MetricsRegistry;
+import org.jboss.netty.channel.Channel;
+
+import java.io.IOException;
+
+public class RequestProcessorSkipCT extends AbstractRequestProcessor {
+
+
+private final ReplyProcessor replyProcessor;
+
+private final LeaseManagement leaseManager;
+private final Panicker panicker;
+private final String tsoHostAndPort;
+
+@Inject
+RequestProcessorSkipCT(MetricsRegistry metrics,
+ TimestampOracle timestampOracle,
+ ReplyProcessor replyProcessor,
+ Panicker panicker,
+ LeaseManagement leaseManager,
+ TSOServerConfig config,
+ LowWatermarkWriter lowWatermarkWriter,
+ String tsoHostAndPort) throws IOException {
+super(metrics, timestampOracle, panicker, config,
lowWatermarkWriter);
+this.replyProcessor = replyProcessor;
+this.tsoHostAndPort = tsoHostAndPort;
+requestRing = disruptor.start();
+this.leaseManager = leaseManager;
+this.panicker = panicker;
+}
+
+private void commitSuicideIfNotMaster() {
+if (!leaseManager.stillInLeasePeriod()) {
+panicker.panic("Replica " + tsoHostAndPort + " lost mastership
whilst flushing data. Committing suicide");
+}
+}
+
+@Override
+public void forwardCommit(long startTimestamp, long commitTimestamp,
Channel c, MonitoringContext monCtx) {
+commitSuicideIfNotMaster();
--- End diff --
Add a comment that his is required since returning commit results when the
TSO is not the leader might violate snapshot isolation. This is because we have
to guarantee that committing transaction from previous tso has to persist all
its data because a new transaction started by a new tso.
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644675#comment-16644675
]
ASF GitHub Bot commented on OMID-90:
Github user ohadshacham commented on a diff in the pull request:
https://github.com/apache/incubator-omid/pull/46#discussion_r22397
--- Diff:
hbase-client/src/test/java/org/apache/omid/transaction/OmidTestBase.java ---
@@ -89,6 +88,8 @@ public void beforeGroups(ITestContext context) throws
Exception {
TSOServerConfig tsoConfig = new TSOServerConfig();
tsoConfig.setPort(1234);
tsoConfig.setConflictMapSize(1000);
+tsoConfig.setLowLatency(isLowLatency());
+tsoConfig.setWaitStrategy("LOW_CPU");
--- End diff --
nice
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...
GitHub user yonigottesman opened a pull request: https://github.com/apache/incubator-omid/pull/46 [OMID-90] Integrate omid low latency to phoenix changes You can merge this pull request into a Git repository by running: $ git pull https://github.com/yonigottesman/incubator-omid phoenix_ll_integration Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-omid/pull/46.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #46 commit 7647733a50431f2b8f5d4495310db59932c46e06 Author: Yonatan Gottesman Date: 2018-10-10T05:49:28Z Integrate omid low latency to phoenix changes ---
