[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/46#discussion_r223974301
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java 
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long 
cellStartTimestamp, long e
 
 // 2) Then check the commit table
 // If the data was written at a previous epoch, check whether 
the transaction was invalidated
-Optional commitTimeStamp = 
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT = 
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
 }
 
 // 3) Read from shadow cell
-commitTimeStamp = 
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp = 
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
 if (commitTimeStamp.isPresent()) {
 return commitTimeStamp.get();
 }
 
+// In case of LL, if found invalid ct cell, still must check 
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
 // 4) Check the epoch and invalidate the entry
--- End diff --

or invalidate in a non latency mode


---

[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/46#discussion_r223993765
  
--- Diff: 
tso-server/src/main/java/org/apache/omid/tso/RequestProcessorSkipCT.java ---
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.tso;
+
+import com.google.inject.Inject;
+import org.apache.omid.metrics.MetricsRegistry;
+import org.jboss.netty.channel.Channel;
+
+import java.io.IOException;
+
+public class RequestProcessorSkipCT extends AbstractRequestProcessor {
+
+
+private final ReplyProcessor replyProcessor;
+
+private final LeaseManagement leaseManager;
+private final Panicker panicker;
+private final String tsoHostAndPort;
+
+@Inject
+RequestProcessorSkipCT(MetricsRegistry metrics,
+   TimestampOracle timestampOracle,
+   ReplyProcessor replyProcessor,
+   Panicker panicker,
+   LeaseManagement leaseManager,
+   TSOServerConfig config,
+   LowWatermarkWriter lowWatermarkWriter,
+   String tsoHostAndPort) throws IOException {
+super(metrics, timestampOracle, panicker, config, 
lowWatermarkWriter);
+this.replyProcessor = replyProcessor;
+this.tsoHostAndPort = tsoHostAndPort;
+requestRing = disruptor.start();
+this.leaseManager = leaseManager;
+this.panicker = panicker;
+}
+
+private void commitSuicideIfNotMaster() {
+if (!leaseManager.stillInLeasePeriod()) {
+panicker.panic("Replica " + tsoHostAndPort + " lost mastership 
whilst flushing data. Committing suicide");
+}
+}
+
+@Override
+public void forwardCommit(long startTimestamp, long commitTimestamp, 
Channel c, MonitoringContext monCtx) {
+commitSuicideIfNotMaster();
--- End diff --

Add a comment that his is required since returning commit results when the 
TSO is not the leader might violate snapshot isolation. This is because we have 
to guarantee that committing transaction from previous tso has to persist all 
its data because a new transaction started by a new tso.


---

[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/46#discussion_r223989230
  
--- Diff: 
transaction-client/src/main/java/org/apache/omid/transaction/AbstractTransactionManager.java
 ---
@@ -350,6 +356,43 @@ private void 
markReadOnlyTransaction(AbstractTransaction readO
 
 }
 
+private void commitLowLatencyTransaction(AbstractTransaction tx)
+throws RollbackException, TransactionException {
+try {
+
+long commitTs = tsoClient.commit(tx.getStartTimestamp(), 
tx.getWriteSet(), tx.getConflictFreeWriteSet()).get();
+boolean committed = 
commitTableWriter.atomicAddCommittedTransaction(tx.getStartTimestamp(),commitTs);
+if (!committed) {
+// Transaction has been invalidated by other client
+rollback(tx);
+
commitTableClient.completeTransaction(tx.getStartTimestamp());
+rolledbackTxsCounter.inc();
+throw new RollbackException("Transaction " + 
tx.getTransactionId() + " got invalidated");
+}
+certifyCommitForTx(tx, commitTs);
+updateShadowCellsAndRemoveCommitTableEntry(tx, postCommitter);
+
+} catch (ExecutionException e) {
+if (e.getCause() instanceof AbortException) { // TSO reports 
Tx conflicts as AbortExceptions in the future
+rollback(tx);
+rolledbackTxsCounter.inc();
+throw new RollbackException("Conflicts detected in tx 
writeset", e.getCause());
+}
+
+if (e.getCause() instanceof ServiceUnavailableException || 
e.getCause() instanceof ConnectionException) {
+errorTxsCounter.inc();
+rollback(tx); // Rollback proactively cause it's likely 
that a new TSOServer is now master
--- End diff --

the leader


---

[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/46#discussion_r223975170
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java 
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long 
cellStartTimestamp, long e
 
 // 2) Then check the commit table
 // If the data was written at a previous epoch, check whether 
the transaction was invalidated
-Optional commitTimeStamp = 
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT = 
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
 }
 
 // 3) Read from shadow cell
-commitTimeStamp = 
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp = 
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
 if (commitTimeStamp.isPresent()) {
 return commitTimeStamp.get();
 }
 
+// In case of LL, if found invalid ct cell, still must check 
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
 // 4) Check the epoch and invalidate the entry
 // if the data was written by a transaction from a previous 
epoch (previous TSO)
-if (cellStartTimestamp < epoch) {
+if (cellStartTimestamp < epoch || isLowLatency) {
 boolean invalidated = 
commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
 if (invalidated) { // Invalid commit timestamp
+
+// If we are running lowLatency Omid, we could have 
manged to invalidate a ct entry,
+// but the committing client already wrote to shadow 
cells:
--- End diff --

This can happen only in low latency mode, since in the regular mode the 
client keeps the commit table entry if persisting the commit was done after the 
tso lost its lease.


---

[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/46#discussion_r223974621
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java 
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long 
cellStartTimestamp, long e
 
 // 2) Then check the commit table
 // If the data was written at a previous epoch, check whether 
the transaction was invalidated
-Optional commitTimeStamp = 
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
-if (commitTimeStamp.isPresent()) {
-return commitTimeStamp.get();
+boolean invalidatedByOther = false;
+Optional commitTimestampFromCT = 
commitTableClient.getCommitTimestamp(cellStartTimestamp).get();
+if (commitTimestampFromCT.isPresent()) {
+if (isLowLatency && !commitTimestampFromCT.get().isValid())
+invalidatedByOther = true;
+else
+return commitTimestampFromCT.get();
 }
 
 // 3) Read from shadow cell
-commitTimeStamp = 
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
+Optional commitTimeStamp = 
readCommitTimestampFromShadowCell(cellStartTimestamp, locator);
 if (commitTimeStamp.isPresent()) {
 return commitTimeStamp.get();
 }
 
+// In case of LL, if found invalid ct cell, still must check 
sc in stage 3 then return
+if (invalidatedByOther) {
+assert(!commitTimestampFromCT.get().isValid());
+return commitTimestampFromCT.get();
+}
+
 // 4) Check the epoch and invalidate the entry
 // if the data was written by a transaction from a previous 
epoch (previous TSO)
-if (cellStartTimestamp < epoch) {
+if (cellStartTimestamp < epoch || isLowLatency) {
 boolean invalidated = 
commitTableClient.tryInvalidateTransaction(cellStartTimestamp).get();
 if (invalidated) { // Invalid commit timestamp
+
+// If we are running lowLatency Omid, we could have 
manged to invalidate a ct entry,
--- End diff --

managed


---

[GitHub] incubator-omid pull request #46: [OMID-90] Integrate omid low latency to pho...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/46#discussion_r223971768
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java 
---
@@ -181,22 +179,44 @@ public CommitTimestamp locateCellCommitTimestamp(long 
cellStartTimestamp, long e
 
 // 2) Then check the commit table
 // If the data was written at a previous epoch, check whether 
the transaction was invalidated
--- End diff --

This "If the data was written at a previous epoch," can be removed.


---

[GitHub] incubator-omid issue #43: [OMID-110]

[ohadshacham]

Github user ohadshacham commented on the issue:

https://github.com/apache/incubator-omid/pull/43
  
+1


---

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206860096
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java
 ---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import com.google.common.base.Optional;
+import com.sun.istack.Nullable;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterBase;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.util.*;
+
+public class TransactionVisibilityFilter extends FilterBase {
+
+// optional sub-filter to apply to visible cells
+private final Filter userFilter;
+private final SnapshotFilterImpl snapshotFilter;
+private final Map shadowCellCache;
+private final HBaseTransaction hbaseTransaction;
+private final Map familyDeletionCache;
--- End diff --

I would add a comment that the row info is redundant in here since reset is 
called between rows and we clear this map in reset.


---

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206861622
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java
 ---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import com.google.common.base.Optional;
+import com.sun.istack.Nullable;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterBase;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.util.*;
+
+public class TransactionVisibilityFilter extends FilterBase {
+
+// optional sub-filter to apply to visible cells
+private final Filter userFilter;
+private final SnapshotFilterImpl snapshotFilter;
+private final Map shadowCellCache;
+private final HBaseTransaction hbaseTransaction;
+private final Map familyDeletionCache;
+
+public SnapshotFilter getSnapshotFilter() {
+return snapshotFilter;
+}
+
+public TransactionVisibilityFilter(@Nullable Filter cellFilter,
+   SnapshotFilterImpl snapshotFilter,
+   HBaseTransaction hbaseTransaction) {
+this.userFilter = cellFilter;
+this.snapshotFilter = snapshotFilter;
+shadowCellCache = new HashMap<>();
+this.hbaseTransaction = hbaseTransaction;
+familyDeletionCache = new HashMap();
+}
+
+@Override
+public ReturnCode filterKeyValue(Cell v) throws IOException {
+if (CellUtils.isShadowCell(v)) {
+Long commitTs =  Bytes.toLong(CellUtil.cloneValue(v));
+shadowCellCache.put(v.getTimestamp(), commitTs);
+// Continue getting shadow cells until one of them fits this 
transaction
+if (hbaseTransaction.getStartTimestamp() >= commitTs) {
+return ReturnCode.NEXT_COL;
+} else {
+return ReturnCode.SKIP;
+}
+} else if (CellUtils.isFamilyDeleteCell(v)) {
+//Delete is part of this transaction
+if (snapshotFilter.getTSIfInTransaction(v, 
hbaseTransaction).isPresent()) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
v.getTimestamp());
+return ReturnCode.NEXT_COL;
+}
+
+if (shadowCellCache.containsKey(v.getTimestamp()) &&
+hbaseTransaction.getStartTimestamp() >= 
shadowCellCache.get(v.getTimestamp())) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
shadowCellCache.get(v.getTimestamp()));
+return ReturnCode.NEXT_COL;
+}
+
+// Try to get shadow cell from region
+final Get get = new Get(CellUtil.cloneRow(v));
+get.setTimeStamp(v.getTimestamp()).setMaxVersions(1);
+get.addColumn(CellUtil.cloneFamily(v), 
CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER));
+Result deleteFamilySC = 
snapshotFilter.getTableAccessWrapper().get(get);
+
+if (!deleteFamilySC.isEmpty() &&
+
Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0] )) < 
hbaseTransaction.getStartTimestamp()){
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0])));
+return ReturnCode.NEXT_COL;
+}
+
+//At last go to com

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206848753
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/OmidSnapshotFilter.java
 ---
@@ -83,92 +76,97 @@ public void start(CoprocessorEnvironment env) throws 
IOException {
 if (commitTableName != null) {
 commitTableConf.setTableName(commitTableName);
 }
-if (commitTableClient == null) {
-commitTableClient = initAndGetCommitTableClient();
-}
-snapshotFilter = new SnapshotFilterImpl(commitTableClient);
-
 LOG.info("Snapshot filter started");
 }
 
 @Override
 public void stop(CoprocessorEnvironment e) throws IOException {
 LOG.info("Stopping snapshot filter coprocessor");
-commitTableClient.close();
+if (snapshotFilterQueue != null) {
+for (SnapshotFilter snapshotFilter: snapshotFilterQueue) {
+
((SnapshotFilterImpl)snapshotFilter).getCommitTableClient().close();
+}
+}
 LOG.info("Snapshot filter stopped");
 }
 
-public void setCommitTableClient(CommitTable.Client commitTableClient) 
{
-this.commitTableClient = commitTableClient;
-this.snapshotFilter.setCommitTableClient(commitTableClient);
-}
 
 @Override
-public void preGetOp(ObserverContext c, 
Get get, List result) throws IOException {
-
-if (get.getAttribute(CellUtils.CLIENT_GET_ATTRIBUTE) == null) 
return;
-
-try {
-get.setAttribute(CellUtils.CLIENT_GET_ATTRIBUTE, null);
-RegionAccessWrapper regionAccessWrapper = new 
RegionAccessWrapper(HBaseShims.getRegionCoprocessorRegion(c.getEnvironment()));
-Result res = regionAccessWrapper.get(get); // get parameters 
were set at the client side
-
-snapshotFilter.setTableAccessWrapper(regionAccessWrapper);
+public void postGetOp(ObserverContext e, 
Get get, List results)
+throws IOException {
+if (get.getFilter() != null) {
+//This get had a filter and used a commit table client that 
must put back
+assert (get.getFilter() instanceof 
TransactionVisibilityFilter);
+TransactionVisibilityFilter filter = 
(TransactionVisibilityFilter)get.getFilter();
+
snapshotFilterQueue.add((SnapshotFilterImpl)filter.getSnapshotFilter());
+}
+}
 
-List filteredKeyValues = Collections.emptyList();
-if (!res.isEmpty()) {
-TSOProto.Transaction transaction = 
TSOProto.Transaction.parseFrom(get.getAttribute(CellUtils.TRANSACTION_ATTRIBUTE));
 
-long id = transaction.getTimestamp();
-long readTs = transaction.getReadTimestamp();
-long epoch = transaction.getEpoch();
-VisibilityLevel visibilityLevel = 
VisibilityLevel.fromInteger(transaction.getVisibilityLevel());
+@Override
+public void preGetOp(ObserverContext e, 
Get get, List results)
+throws IOException {
 
+if (get.getAttribute(CellUtils.CLIENT_GET_ATTRIBUTE) == null) 
return;
 
-HBaseTransaction hbaseTransaction = new 
HBaseTransaction(id, readTs, visibilityLevel, epoch, new 
HashSet(), new HashSet(), null);
-filteredKeyValues = 
snapshotFilter.filterCellsForSnapshot(res.listCells(), hbaseTransaction, 
get.getMaxVersions(), new HashMap(), get.getAttributesMap());
-}
+HBaseTransaction hbaseTransaction = 
getHBaseTransaction(get.getAttribute(CellUtils.TRANSACTION_ATTRIBUTE));
+SnapshotFilterImpl snapshotFilter = getSnapshotFilter(e);
+get.setMaxVersions();
--- End diff --

I would add a comment that we set to max versions since we are doing Omid 
filtering in the VisibilityFilter


---

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206867892
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java
 ---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import com.google.common.base.Optional;
+import com.sun.istack.Nullable;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterBase;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.util.*;
+
+public class TransactionVisibilityFilter extends FilterBase {
+
+// optional sub-filter to apply to visible cells
+private final Filter userFilter;
+private final SnapshotFilterImpl snapshotFilter;
+private final Map shadowCellCache;
+private final HBaseTransaction hbaseTransaction;
+private final Map familyDeletionCache;
+
+public SnapshotFilter getSnapshotFilter() {
+return snapshotFilter;
+}
+
+public TransactionVisibilityFilter(@Nullable Filter cellFilter,
+   SnapshotFilterImpl snapshotFilter,
+   HBaseTransaction hbaseTransaction) {
+this.userFilter = cellFilter;
+this.snapshotFilter = snapshotFilter;
+shadowCellCache = new HashMap<>();
+this.hbaseTransaction = hbaseTransaction;
+familyDeletionCache = new HashMap();
+}
+
+@Override
+public ReturnCode filterKeyValue(Cell v) throws IOException {
+if (CellUtils.isShadowCell(v)) {
+Long commitTs =  Bytes.toLong(CellUtil.cloneValue(v));
+shadowCellCache.put(v.getTimestamp(), commitTs);
+// Continue getting shadow cells until one of them fits this 
transaction
+if (hbaseTransaction.getStartTimestamp() >= commitTs) {
+return ReturnCode.NEXT_COL;
+} else {
+return ReturnCode.SKIP;
+}
+} else if (CellUtils.isFamilyDeleteCell(v)) {
+//Delete is part of this transaction
+if (snapshotFilter.getTSIfInTransaction(v, 
hbaseTransaction).isPresent()) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
v.getTimestamp());
+return ReturnCode.NEXT_COL;
+}
+
+if (shadowCellCache.containsKey(v.getTimestamp()) &&
+hbaseTransaction.getStartTimestamp() >= 
shadowCellCache.get(v.getTimestamp())) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
shadowCellCache.get(v.getTimestamp()));
+return ReturnCode.NEXT_COL;
+}
+
+// Try to get shadow cell from region
+final Get get = new Get(CellUtil.cloneRow(v));
+get.setTimeStamp(v.getTimestamp()).setMaxVersions(1);
+get.addColumn(CellUtil.cloneFamily(v), 
CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER));
+Result deleteFamilySC = 
snapshotFilter.getTableAccessWrapper().get(get);
+
+if (!deleteFamilySC.isEmpty() &&
+
Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0] )) < 
hbaseTransaction.getStartTimestamp()){
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0])));
+return ReturnCode.NEXT_COL;
+}
+
+//At last go to com

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206856182
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java
 ---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import com.google.common.base.Optional;
+import com.sun.istack.Nullable;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterBase;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.util.*;
+
+public class TransactionVisibilityFilter extends FilterBase {
+
+// optional sub-filter to apply to visible cells
+private final Filter userFilter;
+private final SnapshotFilterImpl snapshotFilter;
+private final Map shadowCellCache;
+private final HBaseTransaction hbaseTransaction;
+private final Map familyDeletionCache;
+
+public SnapshotFilter getSnapshotFilter() {
+return snapshotFilter;
+}
+
+public TransactionVisibilityFilter(@Nullable Filter cellFilter,
+   SnapshotFilterImpl snapshotFilter,
+   HBaseTransaction hbaseTransaction) {
+this.userFilter = cellFilter;
+this.snapshotFilter = snapshotFilter;
+shadowCellCache = new HashMap<>();
+this.hbaseTransaction = hbaseTransaction;
+familyDeletionCache = new HashMap();
+}
+
+@Override
+public ReturnCode filterKeyValue(Cell v) throws IOException {
+if (CellUtils.isShadowCell(v)) {
+Long commitTs =  Bytes.toLong(CellUtil.cloneValue(v));
+shadowCellCache.put(v.getTimestamp(), commitTs);
+// Continue getting shadow cells until one of them fits this 
transaction
+if (hbaseTransaction.getStartTimestamp() >= commitTs) {
+return ReturnCode.NEXT_COL;
+} else {
+return ReturnCode.SKIP;
+}
+} else if (CellUtils.isFamilyDeleteCell(v)) {
+//Delete is part of this transaction
+if (snapshotFilter.getTSIfInTransaction(v, 
hbaseTransaction).isPresent()) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
v.getTimestamp());
+return ReturnCode.NEXT_COL;
+}
+
+if (shadowCellCache.containsKey(v.getTimestamp()) &&
--- End diff --

You can do a get and check the existence using the return value. It saves a 
collection access.


---

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206857467
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java
 ---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import com.google.common.base.Optional;
+import com.sun.istack.Nullable;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterBase;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.util.*;
+
+public class TransactionVisibilityFilter extends FilterBase {
+
+// optional sub-filter to apply to visible cells
+private final Filter userFilter;
+private final SnapshotFilterImpl snapshotFilter;
+private final Map shadowCellCache;
+private final HBaseTransaction hbaseTransaction;
+private final Map familyDeletionCache;
+
+public SnapshotFilter getSnapshotFilter() {
+return snapshotFilter;
+}
+
+public TransactionVisibilityFilter(@Nullable Filter cellFilter,
+   SnapshotFilterImpl snapshotFilter,
+   HBaseTransaction hbaseTransaction) {
+this.userFilter = cellFilter;
+this.snapshotFilter = snapshotFilter;
+shadowCellCache = new HashMap<>();
+this.hbaseTransaction = hbaseTransaction;
+familyDeletionCache = new HashMap();
+}
+
+@Override
+public ReturnCode filterKeyValue(Cell v) throws IOException {
+if (CellUtils.isShadowCell(v)) {
+Long commitTs =  Bytes.toLong(CellUtil.cloneValue(v));
+shadowCellCache.put(v.getTimestamp(), commitTs);
+// Continue getting shadow cells until one of them fits this 
transaction
+if (hbaseTransaction.getStartTimestamp() >= commitTs) {
+return ReturnCode.NEXT_COL;
+} else {
+return ReturnCode.SKIP;
+}
+} else if (CellUtils.isFamilyDeleteCell(v)) {
+//Delete is part of this transaction
+if (snapshotFilter.getTSIfInTransaction(v, 
hbaseTransaction).isPresent()) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
v.getTimestamp());
+return ReturnCode.NEXT_COL;
+}
+
+if (shadowCellCache.containsKey(v.getTimestamp()) &&
+hbaseTransaction.getStartTimestamp() >= 
shadowCellCache.get(v.getTimestamp())) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
shadowCellCache.get(v.getTimestamp()));
+return ReturnCode.NEXT_COL;
+}
+
+// Try to get shadow cell from region
+final Get get = new Get(CellUtil.cloneRow(v));
+get.setTimeStamp(v.getTimestamp()).setMaxVersions(1);
+get.addColumn(CellUtil.cloneFamily(v), 
CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER));
--- End diff --

Change the name of addShadowCellSuffix to addShadowCellMetadata?


---

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206862264
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java
 ---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import com.google.common.base.Optional;
+import com.sun.istack.Nullable;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterBase;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.util.*;
+
+public class TransactionVisibilityFilter extends FilterBase {
+
+// optional sub-filter to apply to visible cells
+private final Filter userFilter;
+private final SnapshotFilterImpl snapshotFilter;
+private final Map shadowCellCache;
+private final HBaseTransaction hbaseTransaction;
+private final Map familyDeletionCache;
+
+public SnapshotFilter getSnapshotFilter() {
+return snapshotFilter;
+}
+
+public TransactionVisibilityFilter(@Nullable Filter cellFilter,
+   SnapshotFilterImpl snapshotFilter,
+   HBaseTransaction hbaseTransaction) {
+this.userFilter = cellFilter;
+this.snapshotFilter = snapshotFilter;
+shadowCellCache = new HashMap<>();
+this.hbaseTransaction = hbaseTransaction;
+familyDeletionCache = new HashMap();
+}
+
+@Override
+public ReturnCode filterKeyValue(Cell v) throws IOException {
+if (CellUtils.isShadowCell(v)) {
+Long commitTs =  Bytes.toLong(CellUtil.cloneValue(v));
+shadowCellCache.put(v.getTimestamp(), commitTs);
+// Continue getting shadow cells until one of them fits this 
transaction
+if (hbaseTransaction.getStartTimestamp() >= commitTs) {
+return ReturnCode.NEXT_COL;
+} else {
+return ReturnCode.SKIP;
+}
+} else if (CellUtils.isFamilyDeleteCell(v)) {
+//Delete is part of this transaction
+if (snapshotFilter.getTSIfInTransaction(v, 
hbaseTransaction).isPresent()) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
v.getTimestamp());
+return ReturnCode.NEXT_COL;
+}
+
+if (shadowCellCache.containsKey(v.getTimestamp()) &&
+hbaseTransaction.getStartTimestamp() >= 
shadowCellCache.get(v.getTimestamp())) {
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
shadowCellCache.get(v.getTimestamp()));
+return ReturnCode.NEXT_COL;
+}
+
+// Try to get shadow cell from region
+final Get get = new Get(CellUtil.cloneRow(v));
+get.setTimeStamp(v.getTimestamp()).setMaxVersions(1);
+get.addColumn(CellUtil.cloneFamily(v), 
CellUtils.addShadowCellSuffix(CellUtils.FAMILY_DELETE_QUALIFIER));
+Result deleteFamilySC = 
snapshotFilter.getTableAccessWrapper().get(get);
+
+if (!deleteFamilySC.isEmpty() &&
+
Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0] )) < 
hbaseTransaction.getStartTimestamp()){
+
familyDeletionCache.put(Bytes.toString(CellUtil.cloneFamily(v)), 
Bytes.toLong(CellUtil.cloneValue(deleteFamilySC.rawCells()[0])));
+return ReturnCode.NEXT_COL;
+}
+
+//At last go to com

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206843754
  
--- Diff: 
hbase-common/src/main/java/org/apache/omid/transaction/CellUtils.java ---
@@ -382,13 +385,16 @@ public int hashCode() {
 hasher.putBytes(cell.getRowArray(), cell.getRowOffset(), 
cell.getRowLength());
 hasher.putBytes(cell.getFamilyArray(), cell.getFamilyOffset(), 
cell.getFamilyLength());
 int qualifierLength = cell.getQualifierLength();
+int qualifierOffset = cell.getQualifierOffset();
 if (isShadowCell()) { // Update qualifier length when 
qualifier is shadow cell
 qualifierLength = 
qualifierLengthFromShadowCellQualifier(cell.getQualifierArray(),
 cell.getQualifierOffset(),
 cell.getQualifierLength());
+qualifierOffset = qualifierOffset + 1;
--- End diff --

Will it work when the shadow cell prefix is absent? legacy data.


---

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206852241
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/omid/transaction/TransactionVisibilityFilter.java
 ---
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import com.google.common.base.Optional;
+import com.sun.istack.Nullable;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellUtil;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.filter.Filter;
+import org.apache.hadoop.hbase.filter.FilterBase;
+import org.apache.hadoop.hbase.util.Bytes;
+
+import java.io.IOException;
+import java.util.*;
+
+public class TransactionVisibilityFilter extends FilterBase {
+
+// optional sub-filter to apply to visible cells
+private final Filter userFilter;
+private final SnapshotFilterImpl snapshotFilter;
+private final Map shadowCellCache;
+private final HBaseTransaction hbaseTransaction;
+private final Map familyDeletionCache;
+
+public SnapshotFilter getSnapshotFilter() {
+return snapshotFilter;
+}
+
+public TransactionVisibilityFilter(@Nullable Filter cellFilter,
+   SnapshotFilterImpl snapshotFilter,
+   HBaseTransaction hbaseTransaction) {
+this.userFilter = cellFilter;
+this.snapshotFilter = snapshotFilter;
+shadowCellCache = new HashMap<>();
+this.hbaseTransaction = hbaseTransaction;
+familyDeletionCache = new HashMap();
+}
+
+@Override
+public ReturnCode filterKeyValue(Cell v) throws IOException {
+if (CellUtils.isShadowCell(v)) {
+Long commitTs =  Bytes.toLong(CellUtil.cloneValue(v));
+shadowCellCache.put(v.getTimestamp(), commitTs);
+// Continue getting shadow cells until one of them fits this 
transaction
+if (hbaseTransaction.getStartTimestamp() >= commitTs) {
--- End diff --

Why do we keep ones that committed after the transaction timestamp?


---

[GitHub] incubator-omid pull request #41: [OMID-102] Support for user Filter when usi...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/41#discussion_r206839768
  
--- Diff: 
hbase-client/src/test/java/org/apache/omid/transaction/TestCellUtils.java ---
@@ -99,11 +99,11 @@ public void testShadowCellQualifiers(byte[] 
shadowCellSuffixToTest) throws IOExc
 public void testCorrectMapingOfCellsToShadowCells() throws IOException 
{
 // Create the required data
 final byte[] validShadowCellQualifier =
-com.google.common.primitives.Bytes.concat(qualifier, 
SHADOW_CELL_SUFFIX);
+com.google.common.primitives.Bytes.concat(new byte[1], 
qualifier, SHADOW_CELL_SUFFIX);
--- End diff --

Define SHADOW_CELL_PREFIX to "new byte[1]"  and replace all the new byte[1].


---

[GitHub] incubator-omid pull request #40: [OMID-106] Delete should use write timestam...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/40


---

[GitHub] incubator-omid pull request #39: [OMID-105] When a tentative family deletion...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/39#discussion_r206422618
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java 
---
@@ -395,8 +427,8 @@ private Get createPendingGet(Cell cell, int 
versionCount) throws IOException {
 }
 }
 
-if (isCellInTransaction(cell, transaction, commitCache) ||
-isCellInSnapshot(cell, transaction, commitCache)) {
+if (isCellInTransaction(cell, transaction, 
commitCache).isPresent() ||
+isCellInSnapshot(cell, transaction, 
commitCache).isPresent()) {
--- End diff --

Same as before


---

[GitHub] incubator-omid pull request #39: [OMID-105] When a tentative family deletion...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/39#discussion_r206422025
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/SnapshotFilterImpl.java 
---
@@ -298,26 +291,62 @@ public CommitTimestamp locateCellCommitTimestamp(long 
cellStartTimestamp, long e
 return commitCache;
 }
 
-private void buildFamilyDeletionCache(List rawCells, Map> familyDeletionCache) {
-
+private void buildFamilyDeletionCache(HBaseTransaction transaction, 
List rawCells, Map familyDeletionCache, Map 
commitCache, Map attributeMap) throws IOException {
 for (Cell cell : rawCells) {
 if (CellUtil.matchingQualifier(cell, 
CellUtils.FAMILY_DELETE_QUALIFIER) &&
 CellUtil.matchingValue(cell, 
HConstants.EMPTY_BYTE_ARRAY)) {
-
 String row = Bytes.toString(cell.getRow());
-List cells = familyDeletionCache.get(row);
-if (cells == null) {
-cells = new ArrayList<>();
-familyDeletionCache.put(row, cells);
+String family = Bytes.toString(cell.getFamily());
+String key = row + ":" + family;
+
+if (familyDeletionCache.containsKey(key))
+return;
+
+Optional commitTimeStamp = isCellInSnapshot(cell, 
transaction, commitCache);
+
+if (!commitTimeStamp.isPresent()) {
+commitTimeStamp = isCellInTransaction(cell, 
transaction, commitCache);
 }
 
-cells.add(cell);
+if (commitTimeStamp.isPresent()) {
+familyDeletionCache.put(key, commitTimeStamp.get());
+} else {
+Cell lastCell = cell;
+Map cmtCache;
+boolean foundCommitttedFamilyDeletion = false;
+while (!foundCommitttedFamilyDeletion) {
+
+Get g = createPendingGet(lastCell, 3);
+for (Map.Entry entry : 
attributeMap.entrySet()) {
--- End diff --

I don't know which attribute the user wrote and whether he/she wrote a 
coprocessor that looks on these attributes and do something. 


---

[GitHub] incubator-omid pull request #36: [OMID-72] bug fix, accessed tables should b...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/36


---

[GitHub] incubator-omid pull request #36: [OMID-72] bug fix, accessed tables should b...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/36

[OMID-72] bug fix, accessed tables should be sent to transaction mana…

…ger also for conflict free writes. This is because fences should also 
force conflict free transactions to abort.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-72-bug

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/36.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #36


commit d96ff96e4fa264472ae684bdb1834e791a6d25ef
Author: Ohad Shacham 
Date:   2018-06-03T13:41:45Z

[OMID-72] bug fix, accessed tables should be sent to transaction manager 
also for conflict free writes. This is because fences should also force 
conflict free transactions to abort.




---

[GitHub] incubator-omid pull request #35: [OMID-100] James Taylor's patch to: https:/...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/35


---

[GitHub] incubator-omid pull request #34: [OMID-99] change TestNG version to 6.10.

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/34

[OMID-99] change TestNG version to 6.10.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-99

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/34.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #34






---

[GitHub] incubator-omid pull request #33: [OMID-98] Adding license headers to default...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/33


---

[GitHub] incubator-omid pull request #29: [OMID-44] Since YCSB files are includes in ...

[ohadshacham]

GitHub user ohadshacham reopened a pull request:

https://github.com/apache/incubator-omid/pull/29

[OMID-44] Since YCSB files are includes in OMID a reference in Yahoo …

…is needed in the NOTICE file.

Subject raised at: 
https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-44

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/29.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #29


commit af7acfaa60fc67c1523069be3e6acd4d7c018daf
Author: Ohad Shacham <ohads@...>
Date:   2018-03-26T08:31:37Z

[OMID-44] Since YCSB files are includes in OMID a reference in Yahoo is 
needed in the NOTICE file.
Subject raised at: 
https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E




---

[GitHub] incubator-omid pull request #32: [OMID-97] New MacBooks include a TouchBar. ...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/32


---

[GitHub] incubator-omid pull request #33: [OMID-98] Adding license headers to default...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/33

[OMID-98] Adding license headers to default-omid-server-configuration…

….yml and default-tso-server-benchmark-config.yml.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid ymlLicenseHeader

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/33.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #33






---

[GitHub] incubator-omid pull request #32: [OMID-97] New MacBooks include a TouchBar. ...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/32

[OMID-97] New MacBooks include a TouchBar. This touch bar has a usb l…

…ink that called iBridge. When choosing a network interface Omid 
accidentally chooses this iBridge instead of choosing the Ethernet or the Wifi.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid fixNetworkInterface

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/32.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #32


commit 4ae4bbd52749908fabda0e445577b253bcb4bda2
Author: Ohad Shacham <ohads@...>
Date:   2018-04-24T12:39:59Z

[OMID-97] New MacBooks include a TouchBar. This touch bar has a usb link 
that called iBridge. When choosing a network interface Omid accidentally 
chooses this iBridge instead of choosing the Ethernet or the Wifi.




---

[GitHub] incubator-omid pull request #31: [OMID-96] Enable compactor on all column fa...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/31


---

[GitHub] incubator-omid pull request #30: [OMID-95] Set hbase-1 as the default compil...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/30


---

[GitHub] incubator-omid pull request #29: [OMID-44] Since YCSB files are includes in ...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/29


---

[GitHub] incubator-omid pull request #29: [OMID-44] Since YCSB files are includes in ...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/29

[OMID-44] Since YCSB files are includes in OMID a reference in Yahoo …

…is needed in the NOTICE file.

Subject raised at: 
https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-44

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/29.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #29


commit af7acfaa60fc67c1523069be3e6acd4d7c018daf
Author: Ohad Shacham <ohads@...>
Date:   2018-03-26T08:31:37Z

[OMID-44] Since YCSB files are includes in OMID a reference in Yahoo is 
needed in the NOTICE file.
Subject raised at: 
https://lists.apache.org/thread.html/10fd534d9ea4b0a4574637ab49e5d823a47a864ec27f317f90b14196@%3Cgeneral.incubator.apache.org%3E




---

[GitHub] incubator-omid pull request #30: [OMID-95] Set hbase-1 as the default compil...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/30

[OMID-95] Set hbase-1 as the default compilation profile



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-95

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/30.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #30


commit 505bf4c4a1e16e9b8ebdc62aa45f1028cf832c4c
Author: Ohad Shacham <ohads@...>
Date:   2018-03-26T10:59:40Z

[OMID-95] Set hbase-1 as the default compilation profile




---

[GitHub] incubator-omid pull request #31: [OMID-96] Enable compactor on all column fa...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/31

[OMID-96] Enable compactor on all column families during initializati…

…on. This comes in addition to the option of marking each column family 
using HBase metadata.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-96

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/31.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #31


commit 1c264e3f409d418057f9ab1cc768648357804493
Author: Ohad Shacham <ohads@...>
Date:   2018-03-26T13:07:49Z

[OMID-96] Enable compactor on all column families during initialization. 
This comes in addition to the option of marking each column family using HBase 
metadata.




---

[GitHub] incubator-omid pull request #25: [OMID-93] mark mutation as committed

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/25


---

[GitHub] incubator-omid pull request #26: A bug fix in [OMID-74].

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/26


---

[GitHub] incubator-omid pull request #28: [OMID-94] Tune Omid for Phoenix testing env...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/28

[OMID-94] Tune Omid for Phoenix testing environment.

This commit changes visibility of several function in order to run Omid in 
testing mode from Phoenix testing environment.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-94

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/28.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #28


commit d1b9a5b338429c6135201cff7266dcb670486880
Author: Ohad Shacham <ohads@...>
Date:   2018-03-19T07:48:58Z

[OMID-94] Tune Omid for Phoenix testing environment.
This commit changes visibility of several function in order to run Omid in 
testing mode from Phoenix testing environment.




---

[GitHub] incubator-omid pull request #26: A bug fix in [OMID-74].

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/26

A bug fix in [OMID-74].

The update of the write set is incorrect since the family deletion 
qualifier needs to be added instead of a row marker. Therefore, this commit 
fixes this case.
This is crucial since the write set information is needed for adding shadow 
cells, when transaction successfully commits, and for garbage collection when 
transaction aborts.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-74-reopen

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/26.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #26


commit d6ed73fc4cf34c84a4f283011521a746f257d66e
Author: Ohad Shacham <ohads@...>
Date:   2018-03-15T12:33:57Z

A bug fix in [OMID-74].
The update of the write set is incorrect since the family deletion 
qualifier needs to be added instead of a row marker. Therefore, this commit 
fixes this case.
This is crucial since the write set information is needed for adding shadow 
cells, when transaction successfully commits, and for garbage collection when 
transaction aborts.




---

[GitHub] incubator-omid pull request #27: reopen [OMID-70] in order to bind WorldCloc...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/27

reopen [OMID-70] in order to bind WorldClockOracleImpl in TSOMockModule.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-70-reopen

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/27.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #27


commit 0eca99f9911a0e23fa099531c59a231098b97fa5
Author: Ohad Shacham <ohads@...>
Date:   2018-03-15T14:22:35Z

reopen [OMID-70] in order to bind WorldClockOracleImpl in TSOMockModule.




---

[GitHub] incubator-omid pull request #25: [OMID-93] mark mutation as committed

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/25

[OMID-93] mark mutation as committed

This pull request adds an option to add commit metadata (shadow cells) to 
an existing mutation. This feature is required by Apache Phoenix both for local 
index population and update.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-93

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/25.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #25


commit 93b2c2a1e7f2e3937feb6a3a507899af13c7c38c
Author: Ohad Shacham <ohads@...>
Date:   2018-03-13T12:22:39Z

[OMID-93] This commit adds an option to add commit metadata (shadow cells) 
to an existing mutation. This feature is required by Apache Phoenix both for 
local index population and update.

commit 361a41373b81c9efa9e77e70c1c58dcdc3b6deb4
Author: Ohad Shacham <ohads@...>
Date:   2018-03-13T12:33:36Z

cosmetics




---

[GitHub] incubator-omid pull request #24: [OMID-92] Add Apache Rat to the build proce...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/24


---

[GitHub] incubator-omid pull request #24: [OMID-92] Add Apache Rat to the build proce...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/24

[OMID-92] Add Apache Rat to the build process and align code with Rat…

…'s requirement.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-92

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/24.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #24


commit aea8b149366264da01a883c616b34675dc74d4bf
Author: Ohad Shacham <ohads@...>
Date:   2018-03-12T13:04:27Z

[OMID-92] Add Apache Rat to the build process and align code with Rat's 
requirement.




---

[GitHub] incubator-omid pull request #23: [OMID-89] Fix metrics in Persistence proces...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/23#discussion_r167857492
  
--- Diff: 
tso-server/src/test/java/org/apache/omid/tso/TestPersistenceProcessorHandler.java
 ---
@@ -126,6 +126,36 @@ void afterMethod() {
 Mockito.reset(mockWriter);
 }
 
+@Test(timeOut = 1_000)
+public void testPersistentProcessorHandlerIdsAreCreatedConsecutive() 
throws Exception {
+
+TSOServerConfig tsoConfig = new TSOServerConfig();
+tsoConfig.setNumConcurrentCTWriters(32);
+
+PersistenceProcessorHandler[] handlers = new 
PersistenceProcessorHandler[tsoConfig.getNumConcurrentCTWriters()];
+for (int i = 0; i < tsoConfig.getNumConcurrentCTWriters(); i++) {
+handlers[i] = new PersistenceProcessorHandler(metrics,
+  "localhost:1234",
+  
mock(LeaseManager.class),
+  commitTable,
+  
mock(ReplyProcessor.class),
+  retryProcessor,
+  panicker);
+}
+
+for (int i = 0; i < tsoConfig.getNumConcurrentCTWriters(); i++) {
+// Required to generalize the cases when other tests have 
increased the static variable assigning the ids
+if (i + 1 < tsoConfig.getNumConcurrentCTWriters()) {
+int followingHandlerIdAsInt = Integer.valueOf(handlers[i + 
1].getId());
+assertEquals(handlers[i].getId(), 
String.valueOf(followingHandlerIdAsInt - 1));
--- End diff --

We kind of testing the atomic integer :)


---

[GitHub] incubator-omid pull request #17: [OMID-83] Attributes added to Put, Get, and...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/17


---

[GitHub] incubator-omid pull request #20: [OMID-85] Writing directly to HBase using s...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/20


---

[GitHub] incubator-omid pull request #20: [OMID-85] Writing directly to HBase using s...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/20

[OMID-85] Writing directly to HBase using specific version marks the …

…write as a write that was done by a specific transaction.

However, due to lack of shadow cells, getting the commit timestamp of the 
transaction can be done only by access the commit table.
The motivation of this feature is to add the shadow cells during the write 
and save the commit table access.
This feature is required by Apache Phoenix that during index creation adds 
the data table's entries,
appeared before creation, to the index. In this case, the version and the 
commit timestamp should be the fence id and therefore, a direct write to HBase 
with the addition of shadow cells is required.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid AutoCommit

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/20.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20


commit 462984833997c6192ae30788f5f7a3d866d41dce
Author: Ohad Shacham <ohads@...>
Date:   2018-02-01T15:13:08Z

[OMID-85] Writing directly to HBase using specific version marks the write 
as a write that was done by a specific transaction.
However, due to lack of shadow cells, getting the commit timestamp of the 
transaction can be done only by access the commit table.
The motivation of this feature is to add the shadow cells during the write 
and save the commit table access.
This feature is required by Apache Phoenix that during index creation adds 
the data table's entries,
appeared before creation, to the index. In this case, the version and the 
commit timestamp should be the fence id and therefore, a direct write to HBase 
with the addition of shadow cells is required.




---

[GitHub] incubator-omid pull request #19: [OMID-84] Today, all the writes done by a t...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/19

[OMID-84] Today, all the writes done by a transaction are taking part…

… in conflict analysis. The purpose of this feature is to let the user 
decide for each write, whether it should take part in the conflict analysis.

The motivation infers from Apache Phoenix that utilizes this feature when 
writing to the secondary index and also when writing to the data table for 
immutable tables (each key is added once and is not modified).

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid OMID-84-from-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/19.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #19


commit 8fad7b57e3bb6ce6b69e753252c43c33af5a8eeb
Author: Ohad Shacham <ohads@...>
Date:   2018-01-31T14:52:43Z

[OMID-84] Today, all the writes done by a transaction are taking part in 
conflict analysis. The purpose of this feature is to let the user decide for 
each write, whether it should take part in the conflict analysis.
The motivation infers from Apache Phoenix that utilizes this feature when 
writing to the secondary index and also when writing to the data table for 
immutable tables (each key is added once and is not modified).




---

[GitHub] incubator-omid pull request #17: [OMID-83] Attributes added to Put, Get, and...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/17

[OMID-83] Attributes added to Put, Get, and Scan are not propagated t…

…o HBase. In many cases, as in the Phoenix case, these attributes are 
required and should be propagated to the server side. In Phoenix for example, 
attributes are used to mark data as one that should propagate to the secondary 
index.

This commit propagates the attributes to HBase side.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid AttributesSupport

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/17.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #17


commit 2bdf776bbb9bbf0e70dd49bec41239f6fcaf654c
Author: Ohad Shacham <oh...@yahoo-inc.com>
Date:   2017-12-19T08:24:26Z

[OMID-83] Attributes added to Put, Get, and Scan are not propagated to 
HBase. In many cases, as in the Phoenix case, these attributes are required and 
should be propagated to the server side. In Phoenix for example, attributes are 
used to mark data as one that should propagate to the secondary index.
This commit propagates the attributes to HBase side.




---

[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...

[ohadshacham]

Github user ohadshacham closed the pull request at:

https://github.com/apache/incubator-omid/pull/16


---

[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/16#discussion_r150795077
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/regionserver/RegionAccessWrapper.java
 ---
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.omid.transaction.TableAccessWrapper;
+
+public class RegionAccessWrapper implements TableAccessWrapper {
--- End diff --

done


---

[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/16#discussion_r150794291
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/regionserver/OmidRegionScanner.java
 ---
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.regionserver.ScannerContext;
+import org.apache.omid.transaction.HBaseTransaction;
+import org.apache.omid.transaction.SnapshotFilter;
+
+public class OmidRegionScanner implements RegionScanner {
+
+RegionScanner scanner;
+SnapshotFilter snapshotFilter;
+HBaseTransaction transaction;
+int maxVersions;
+Map<String, List> familyDeletionCache;
+
+public OmidRegionScanner(SnapshotFilter snapshotFilter,
+  RegionScanner s,
+  HBaseTransaction transaction,
+  int maxVersions) {
+this.snapshotFilter = snapshotFilter;
+this.scanner = s;
+this.transaction = transaction;
+this.maxVersions = maxVersions;
+this.familyDeletionCache = new HashMap<String, List>();
+}
+
+@Override
+public boolean next(List results) throws IOException {
+   return next(results, Integer.MAX_VALUE);
+}
+
+public boolean next(List result, int limit) throws IOException {
--- End diff --

The function is defined in RegionScanner interface, we are just override it.


---

[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/16#discussion_r150794375
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/HTableAccessWrapper.java 
---
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.omid.transaction;
+
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.HTableInterface;
+import org.apache.hadoop.hbase.client.Put;
+import org.apache.hadoop.hbase.client.Result;
+
+public class HTableAccessWrapper implements TableAccessWrapper {
--- End diff --

RegionAccessWrapper


---

[GitHub] incubator-omid pull request #16: [OMID-78] Identify transaction snapshot at ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/16#discussion_r150794314
  
--- Diff: 
hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/regionserver/OmidRegionScanner.java
 ---
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.regionserver.ScannerContext;
+import org.apache.omid.transaction.HBaseTransaction;
+import org.apache.omid.transaction.SnapshotFilter;
+
+public class OmidRegionScanner implements RegionScanner {
+
+RegionScanner scanner;
+SnapshotFilter snapshotFilter;
+HBaseTransaction transaction;
+int maxVersions;
+Map<String, List> familyDeletionCache;
+
+public OmidRegionScanner(SnapshotFilter snapshotFilter,
+  RegionScanner s,
+  HBaseTransaction transaction,
+  int maxVersions) {
+this.snapshotFilter = snapshotFilter;
+this.scanner = s;
+this.transaction = transaction;
+this.maxVersions = maxVersions;
+this.familyDeletionCache = new HashMap<String, List>();
+}
+
+@Override
+public boolean next(List results) throws IOException {
+   return next(results, Integer.MAX_VALUE);
+}
+
+public boolean next(List result, int limit) throws IOException {
+return nextRaw(result, limit);
+}
+
+@Override
+public void close() throws IOException {
+scanner.close();
+}
+
+@Override
+public HRegionInfo getRegionInfo() {
+return scanner.getRegionInfo();
+}
+
+@Override
+public boolean isFilterDone() throws IOException {
+return scanner.isFilterDone();
+}
+
+@Override
+public boolean reseek(byte[] row) throws IOException {
+throw new RuntimeException("Not implemented");
+}
+
+@Override
+public long getMaxResultSize() {
+return scanner.getMaxResultSize();
+}
+
+@Override
+public long getMvccReadPoint() {
+return scanner.getMvccReadPoint();
+}
+
+@Override
+public boolean nextRaw(List result) throws IOException {
--- End diff --

The function is defined in RegionScanner interface, we are just override it.


---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130809257
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -396,20 +443,48 @@ public ResultScanner getScanner(Transaction tx, Scan 
scan) throws IOException {
 return commitCache;
 }
 
-private boolean isCellInSnapshot(Cell kv, HBaseTransaction 
transaction, Map<Long, Long> commitCache)
-throws IOException {
+private void buildFamilyDeletionCache(List rawCells, Map<String, 
List> familyDeletionCache) {
 
-long startTimestamp = transaction.getStartTimestamp();
+for (Cell cell : rawCells) {
+if (CellUtil.matchingQualifier(cell, 
CellUtils.FAMILY_DELETE_QUALIFIER) &&
+CellUtil.matchingValue(cell, 
HConstants.EMPTY_BYTE_ARRAY)) {
+
+String row = Bytes.toString(cell.getRow());
+List cells = familyDeletionCache.get(row);
--- End diff --

This is Map's return value.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ohadshacham]

Github user ohadshacham commented on a diff in the pull request:

https://github.com/apache/incubator-omid/pull/13#discussion_r130805885
  
--- Diff: 
hbase-client/src/main/java/org/apache/omid/transaction/TTable.java ---
@@ -228,21 +261,12 @@ public void delete(Transaction tx, Delete delete) 
throws IOException {
 }
 }
 }
-if (issueGet) {
-// It's better to perform a transactional get to avoid 
deleting more
-// than necessary
-Result result = this.get(transaction, deleteG);
-if (!result.isEmpty()) {
-for (Entry<byte[], NavigableMap<byte[], NavigableMap<Long, 
byte[]>>> entryF : result.getMap()
-.entrySet()) {
-byte[] family = entryF.getKey();
-for (Entry<byte[], NavigableMap<Long, byte[]>> entryQ 
: entryF.getValue().entrySet()) {
-byte[] qualifier = entryQ.getKey();
-deleteP.add(family, qualifier, 
CellUtils.DELETE_TOMBSTONE);
-transaction.addWriteSetElement(new 
HBaseCellId(table, delete.getRow(), family, qualifier,
-   
transaction.getStartTimestamp()));
-}
-}
+
+if (deleteFamily) {
--- End diff --

1. This is a deletion path without read requirement. It is doable only in 
row level conflict detection mode where we only need to mark the row as a 
player for conflict analysis. 
2. TTable works only for HBase. You can see that each one of the method 
starts with a check that the transaction is hbase transaction. IMHO TTable 
should inherit from HTableInterface and override its functions, but this is for 
 a different patch :).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-omid pull request #13: [OMID-74] Efficient column family deletion ...

[ohadshacham]

GitHub user ohadshacham opened a pull request:

https://github.com/apache/incubator-omid/pull/13

[OMID-74] Efficient column family deletion in Row level conflict analysis

The idea is to use a qualifier to denote that all the columns of a specific 
family were deleted.
Current implementation reads from HBase the entire family and then writes a 
tombstone to each one of its cells.
The new implementation does not need to perform the read and only writes 
the qualifier to denote that the family was deleted.
This is true only for Row level conflict detection since in Cell level we 
need to read the cells and add these to the write set.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ohadshacham/incubator-omid 
FamilyDeletionTombstone-squash

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-omid/pull/13.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13


commit 35ba4c79b4d3f713c6c38f791d5898f778153e2a
Author: Ohad Shacham <oh...@yahoo-inc.com>
Date:   2017-07-27T13:12:31Z

[OMID-74] Efficient column family deletion in Row level conflict analysis
The idea is to use a qualifier to denote that all the columns of a specific 
family were deleted.
Current implementation reads from HBase the entire family and then writes a 
tombstone to each one of its cells.
The new implementation does not need to perform the read and only writes 
the qualifier to denote that the family was deleted.
This is true only for Row level conflict detection since in Cell level we 
need to read the cells and add these to the write set.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---