[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654279#comment-16654279
]
James Taylor commented on OMID-90:
--
{quote}Where would one go to edit stuff like the command used to compile and
stuff like that?
{quote}
Good question. Unfortunately I don't know either. [~mujtabachohan] has always
graciously made these changes. However, since our builds are working now, can't
you just check-in something to Omid's phoenix-integration branch to make low
latency the default and then kick off an omid2 Jenkins job? And then just
revert it back.
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654145#comment-16654145 ] Yonatan Gottesman commented on OMID-90: --- Hi, [~jamestaylor] I dont have any experience with editing jenkins jobs. Where would one go to edit stuff like the command used to compile and stuff like that? > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654068#comment-16654068 ] Ohad Shacham commented on OMID-90: -- Awesome, Thx. > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654048#comment-16654048 ] Ohad Shacham commented on OMID-90: -- I assume that [~yonigo] can commit the low latency, we already reviewed everything. He wanted to wait until everything passes before he does the commit. The low latency is not default, [~jamestaylor] how can we change the configuration when running the jenkins? Do we have permissions to do some? > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653655#comment-16653655 ] Yonatan Gottesman commented on OMID-109: Hi i fixed this and wil upload a patch later today, so don’t wast your time on it > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653654#comment-16653654 ] James Taylor commented on OMID-90: -- Ok, sounds like we're good, then. Is it ok if [~yonigo] commits this now, [~ohads]? Would it be possible to get a Phoenix test run on the omid2 branch with the low latency version? > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[
https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653649#comment-16653649
]
James Taylor commented on OMID-109:
---
Does it fix it if you add the following dependency to benchmarks/pom.xml?
{code:java}
hbase-1
org.apache.omid
omid-hbase1-shims
${project.version}
hbase-2
org.apache.omid
omid-hbase2-shims
${project.version}
{code}
> Unable to build phoenix-integration branch through Jenkins job
> --
>
> Key: OMID-109
> URL: https://issues.apache.org/jira/browse/OMID-109
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Blocker
> Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch
>
>
> Based on Jenkins job failures
> (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom
> needs to be updated to
> https://raw.githubusercontent.com/synergian/wagon-git/releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653382#comment-16653382 ] Yonatan Gottesman commented on OMID-109: [~jamestaylor] Im having an issue with building hbase-2 I haven't seen before: If you remove all omid local installations from ~/.m2/org/apache/omid and then try to mvn clean packge -DskipTests -Phbase-2 This fails while compiling benchmarks because the maven assembly plugin is looking for the shims1 (whitch isnt compiled). We missed this because if you run mvn clean install hbase-1 before then the jar gets in your local m2 directory and then the assembly finds it. Anyway can you try "mvn clean package -Phbase-2" on a clean env after deleting all omid installations in m2? > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653340#comment-16653340
]
Ohad Shacham commented on OMID-90:
--
{quote}In PHOENIX-4943, I write the shadow cells of the index rows when the
index rows themselves are written. Does that solve the issue?
{quote}
Yes [~jamestaylor], the fence functionality exists in the transaction manager.
The thing is that since during index population Phoenix writes that rows with
the fence id, then before the fence id is not written to the commit table, we
have to guarantee that the rows are written with the shadow cells. Otherwise,
there might be a case that someone will try to read before the shadow cells
exists. On a seconds thought, maybe it is not possible since the index is not
valid yet? In any case, we want to do it since writing once is better than
writing and then writing again the shadow cells.
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor resolved OMID-117. --- Resolution: Fixed > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_addendum1.patch, > OMID-117_hbase2.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch, OMID-117_v6.patch, OMID-117_v7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653047#comment-16653047 ] Yonatan Gottesman commented on OMID-109: It looks like the TSO is trying to bind to an address that is already used, maybe by a previous TSO from previous test. Ill look into it also > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[
https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653024#comment-16653024
]
James Taylor commented on OMID-109:
---
Also, I'm seeing the following error sometimes:
{code:java}
[ERROR] Tests run: 56, Failures: 0, Errors: 4, Skipped: 8, Time elapsed:
317.357 s <<< FAILURE! - in
org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT
[ERROR]
testRowCountAndByteCounts[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.841 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
[ERROR]
testSomeUpdateEmptyStats[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.117 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
[ERROR]
testUpdateStatsWithMultipleTables[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.101 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
[ERROR]
testUpdateStats[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.1 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
{code}
Any ideas?
> Unable to build phoenix-integration branch through Jenkins job
> --
>
> Key: OMID-109
> URL: https://issues.apache.org/jira/browse/OMID-109
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Blocker
> Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch
>
>
> Based on Jenkins job failures
> (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom
> needs to be updated to
> https://raw.githubusercontent.com/synergian/wagon-git/releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
