[jira] [Commented] (OMID-136) Apache Incubator report - March 2019
[ https://issues.apache.org/jira/browse/OMID-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783532#comment-16783532 ] James Taylor commented on OMID-136: --- +1. Report looks good. > Apache Incubator report - March 2019 > > > Key: OMID-136 > URL: https://issues.apache.org/jira/browse/OMID-136 > Project: Apache Omid > Issue Type: Task >Reporter: Ohad Shacham >Assignee: Ohad Shacham >Priority: Major > > Provide a report for December 1, 2018 - February 28, 2019 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-119) OMID-119 Generate different jar names for hbase1 and hbase2
[ https://issues.apache.org/jira/browse/OMID-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16662507#comment-16662507 ] James Taylor commented on OMID-119: --- +1. I checked this in. Thanks, [~yonigo]. > OMID-119 Generate different jar names for hbase1 and hbase2 > --- > > Key: OMID-119 > URL: https://issues.apache.org/jira/browse/OMID-119 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Yonatan Gottesman >Assignee: Yonatan Gottesman >Priority: Major > Fix For: 1.0.0.0 > > Attachments: omid-119_v2.patch, omid119.patch, omid_versions.patch > > > For the release we need to have a different jar for each hbase version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659805#comment-16659805 ] James Taylor commented on OMID-109: --- [~yonigo] - do you need me to check-in the omidll.patch to Phoenix? Does it address the test failures you've seen? I don't have an idea about those failures (as I mentioned in my comment [here|https://issues.apache.org/jira/browse/OMID-109?focusedCommentId=16653024=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16653024]). > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Fix For: 1.0.0.0 > > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch, > omid90.patch, omidll.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-118) Update pom version to latest Omid version
[ https://issues.apache.org/jira/browse/OMID-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-118: -- Attachment: OMID-118.patch > Update pom version to latest Omid version > - > > Key: OMID-118 > URL: https://issues.apache.org/jira/browse/OMID-118 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-118.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OMID-118) Update pom version to latest Omid version
James Taylor created OMID-118: - Summary: Update pom version to latest Omid version Key: OMID-118 URL: https://issues.apache.org/jira/browse/OMID-118 Project: Apache Omid Issue Type: Sub-task Reporter: James Taylor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655463#comment-16655463 ] James Taylor commented on OMID-109: --- Everything thing is good with the build, [~yonigo]. There are existing flappers in the unit tests, unrelated to our work. You mentioned you have a patch for hbase-2 which I haven’t seen yet? > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655320#comment-16655320 ] James Taylor commented on OMID-90: -- Everything is good with OMID-109. All the transaction tests pass. Last run the failure was a known flapper, unrelated to our work. > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16654279#comment-16654279
]
James Taylor commented on OMID-90:
--
{quote}Where would one go to edit stuff like the command used to compile and
stuff like that?
{quote}
Good question. Unfortunately I don't know either. [~mujtabachohan] has always
graciously made these changes. However, since our builds are working now, can't
you just check-in something to Omid's phoenix-integration branch to make low
latency the default and then kick off an omid2 Jenkins job? And then just
revert it back.
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653654#comment-16653654 ] James Taylor commented on OMID-90: -- Ok, sounds like we're good, then. Is it ok if [~yonigo] commits this now, [~ohads]? Would it be possible to get a Phoenix test run on the omid2 branch with the low latency version? > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[
https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653649#comment-16653649
]
James Taylor commented on OMID-109:
---
Does it fix it if you add the following dependency to benchmarks/pom.xml?
{code:java}
hbase-1
org.apache.omid
omid-hbase1-shims
${project.version}
hbase-2
org.apache.omid
omid-hbase2-shims
${project.version}
{code}
> Unable to build phoenix-integration branch through Jenkins job
> --
>
> Key: OMID-109
> URL: https://issues.apache.org/jira/browse/OMID-109
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Blocker
> Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch
>
>
> Based on Jenkins job failures
> (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom
> needs to be updated to
> https://raw.githubusercontent.com/synergian/wagon-git/releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor resolved OMID-117. --- Resolution: Fixed > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_addendum1.patch, > OMID-117_hbase2.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch, OMID-117_v6.patch, OMID-117_v7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[
https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653024#comment-16653024
]
James Taylor commented on OMID-109:
---
Also, I'm seeing the following error sometimes:
{code:java}
[ERROR] Tests run: 56, Failures: 0, Errors: 4, Skipped: 8, Time elapsed:
317.357 s <<< FAILURE! - in
org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT
[ERROR]
testRowCountAndByteCounts[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.841 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
[ERROR]
testSomeUpdateEmptyStats[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.117 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
[ERROR]
testUpdateStatsWithMultipleTables[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.101 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
[ERROR]
testUpdateStats[mutable=false,transactionProvider=OMID,isUserTableNamespaceMapped=false,columnEncoded=false](org.apache.phoenix.end2end.NonColumnEncodedImmutableTxStatsCollectorIT)
Time elapsed: 0.1 s <<< ERROR!
com.google.common.util.concurrent.UncheckedExecutionException:
org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to:
0.0.0.0/0.0.0.0:35726
Caused by: java.net.BindException: Address already in use
{code}
Any ideas?
> Unable to build phoenix-integration branch through Jenkins job
> --
>
> Key: OMID-109
> URL: https://issues.apache.org/jira/browse/OMID-109
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Blocker
> Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch
>
>
> Based on Jenkins job failures
> (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom
> needs to be updated to
> https://raw.githubusercontent.com/synergian/wagon-git/releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652692#comment-16652692 ] James Taylor commented on OMID-117: --- [~yonigo] - please review my addendum patch. I lowered the default number of read timeouts to one - this seems to have fixed the issue and wouldn't be an issue in production. Also, worst case, we could always override the default (without needing a release) to be larger if we encounter issues. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_addendum1.patch, > OMID-117_hbase2.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch, OMID-117_v6.patch, OMID-117_v7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_addendum1.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_addendum1.patch, > OMID-117_hbase2.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch, OMID-117_v6.patch, OMID-117_v7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652519#comment-16652519 ] James Taylor commented on OMID-90: -- FYI, the test that inserts rows as the index is being initially populated is BaseIndexIT.testCreateIndexAfterUpsertStarted(). That'll be tested when GlobalMutableTxIndexIT is run. Would be good to try this with the low latency version of Omid. Even better would be to try a complete run with the low latency version. > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652287#comment-16652287 ] James Taylor commented on OMID-109: --- If [~ohads] is ok with committing omid low latency, I as as well. Will that commit change the default to use low latency? Would be good to do a Phoenix omid2 jenkins run using the low latency version. Now that the build is working again, do any of these configuration properties that you have Mujtaba set need to be folded into the pom in some way (so it'll build once it gets committed to Phoenix)? > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651850#comment-16651850 ] James Taylor commented on OMID-90: -- In PHOENIX-4943, I write the shadow cells of the index rows when the index rows themselves are written. Does that solve the issue? The fence was more to guard against the case in which a write to the data table occurs before we start maintaining the index. Prior to PHOENIX-4943, we were writing the shadow cells in the standard way (i.e. after the commit succeeded). There's a test for this - would be good if you guys ran the Phoenix unit tests with the low latency version. I don't remember the particular test that tests writing to a table while the index is being created - [~tdsilva] - do you remember? > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651825#comment-16651825 ] James Taylor commented on OMID-109: --- Thanks for offering to help. I'm fixing some of the obvious ones. Main cause is that I've made it so that local indexes are not allowed to be used with Omid, but some tests are still expecting that they would work. Other tests are failing because the shadow cells are throwing off expected number of Cells and/or byte sizes. Let's see how things look after my next commit. Will let you know if I need help. > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650580#comment-16650580 ] James Taylor commented on OMID-109: --- [~yonigo] & [~ohads] - would you guys mind filing INFRA Jira tickets so that you can make changes to these Jenkins jobs? > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[
https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650361#comment-16650361
]
James Taylor commented on OMID-90:
--
Ping, [~ohads]. I didn't understand your question:
{quote}This commit removes the addition of the fence information to the commit
table. Could you please check if it hurts correctness? It is related to the
index population with auto commit that you have just added.
{quote}
We need the fence functionality in Phoenix so that when an index is created, we
can guarantee that we don't miss creation of any data rows.
> Reducing begin/commit latency by distributing the write to the commit table
> ---
>
> Key: OMID-90
> URL: https://issues.apache.org/jira/browse/OMID-90
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: Ohad Shacham
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OmidCloud-VLDB.pdf, omid90.patch
>
>
> Today, Omid's commits are done by the transaction manager. In order to
> efficiently write to the commit table, the transaction manager batches these
> writes. This optimization, even thought reduces the write time to HBase,
> significantly increases the begin and commit latency. The commit latency
> increases since a commit operation returns only after its commit timestamp
> was persisted in the commit table. And the begin latency increases since
> begin returns a transaction id that is also used by the transaction to
> identify its snapshot and therefore, begin returns only after all commits
> with commit id smaller than the begin id was persisted in the commit table.
> This is crucial, since a snapshot change during a transaction run may violate
> snapshot isolation.
>
> The idea of this feature is to distribute the commit by moving the write to
> the commit table from the server to the client. The transaction manager does
> conflict analysis and returns a commit timestamp. While the client atomically
> persists this commit in the commit table.
> This significantly reduces the begin and commit latency, since batching is
> not required anymore. A begin operation can immediately returns and a commit
> operation returns after conflict detection.
> This can introduce snapshot isolation violation since a slow client can
> commit and change other transaction's snapsho. Therefore, we use an
> invalidation technique which is similar to the one Omid uses today to
> maintain snapshot isolation in high availability mode.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[
https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650343#comment-16650343
]
James Taylor commented on OMID-117:
---
{quote}About the retries, what the worst thing that can happen with this? how
bad is it to have it like this?
{quote}
Check out the comment I added to RegionConnectionFactory:
{code:java}
// This setting controls how many retries occur on the region server if an
// IOException occurs while trying to access the commit table. Because a
// handler thread will be in use while these retries occur and the client
// will be blocked waiting, it must not tie up the call for longer than
// the client RPC timeout. Otherwise, the client will initiate retries on it's
// end, tying up yet another handler thread. It's best if the retries can be
// zero, as in that case the handler is released and the retries occur on the
// client side. In testing, we've seen NoServerForRegionException occur which
// is a DoNotRetryIOException which are not retried on the client. It's not
// clear if this is a real issue or a test-only issue.
private static final int DEFAULT_COMMIT_TABLE_ACCESS_ON_READ_RETRIES_NUMBER =
11;
private static final int DEFAULT_COMMIT_TABLE_ACCESS_ON_READ_RETRY_PAUSE = 100;
{code}
As it is with this patch, if retries are necessary to reach the RS hosting the
commit table, they will occur from the RS handling the scan for 48 seconds.
During this time, the handler thread will be tied up (i.e. it won't be able to
be used by any other HBase client). If this occurs for all the handler threads
on the RS, then all incoming requests would be queued. For example, non
transactional queries would potentially not be processed during this time. If
the retries (and pauses) occur on the client side, then non transactional work
loads wouldn't be impacted.
Ideally, we'd have a test that reproduces this NoServerForRegionException and
see if any changes are needed to handle this situation. You might be able to
repro this by manually splitting the commit table and then performing a read
against a transactional table. It also may just occur the very first time the
commit table is attempted to be reached from a RS after the commit table is
created.
> Ensure timeouts are configured low for RPCs to commit table
> ---
>
> Key: OMID-117
> URL: https://issues.apache.org/jira/browse/OMID-117
> Project: Apache Omid
> Issue Type: Bug
>Reporter: James Taylor
>Priority: Major
> Attachments: OMID-117.patch, OMID-117_hbase2.patch,
> OMID-117_v2.patch, OMID-117_v3.patch, OMID-117_v4.patch, OMID-117_v5.patch,
> OMID-117_v6.patch, OMID-117_v7.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_hbase2.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_hbase2.patch, > OMID-117_v2.patch, OMID-117_v3.patch, OMID-117_v4.patch, OMID-117_v5.patch, > OMID-117_v6.patch, OMID-117_v7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649618#comment-16649618 ] James Taylor commented on OMID-117: --- The InterRegionServerRpcController is only a few lines of code and you'd need the constructor in both classes, so I don't think it's worth it. I've attached a v7 that passes for hbase-1 and hbase-2. For the tests to pass, I had to have the server-side retry. Without this, we'd get a NoServerForRegionException which is a DoNotRetryIOException so doesn't trigger any retries on the client. I'm not sure if this is a test-only issue. I added a long comment, but I don't have the cycles to explore further. For hbase-2 I could only test it by changing the poms to make hbase-2 the default profile (see attached). I'll let that be figured out for OMID-109. Is this ok to commit, [~yonigo]? > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch, OMID-117_v6.patch, OMID-117_v7.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648987#comment-16648987 ] James Taylor commented on OMID-117: --- The patch applied to the head of the phoenix-integration branch (which includes 116 as it's checked in already). > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch, OMID-117_v6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OMID-116) Add addShadowCells argument to other TTable methods
[ https://issues.apache.org/jira/browse/OMID-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor resolved OMID-116. --- Resolution: Fixed > Add addShadowCells argument to other TTable methods > --- > > Key: OMID-116 > URL: https://issues.apache.org/jira/browse/OMID-116 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-116.patch > > > We need to pass through the addShadowCells are to all the TTable methods that > can do puts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_v6.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch, OMID-117_v6.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648289#comment-16648289 ] James Taylor commented on OMID-117: --- Please hold off on reviewing the latest patch - I found an issue for hbase-2. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647975#comment-16647975 ] James Taylor commented on OMID-117: --- Ping [~yonigo] or [~ohads]. Ok to commit now? > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647183#comment-16647183 ] James Taylor commented on OMID-109: --- Mujtaba made the change you requested. He thinks you need to file a INFRA Jira to get access to Apache Jenkins builds. > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646999#comment-16646999 ] James Taylor commented on OMID-109: --- I don't know how to do this, [~yonigo], but I've asked [~mujtabachohan] for help. > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch, omid109_v2.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646680#comment-16646680 ] James Taylor commented on OMID-90: -- If you could please commit the smaller OMID-117 before this one to save any merge headaches, that be appreciated. > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646674#comment-16646674 ] James Taylor commented on OMID-117: --- I've attached a v5 that fixes the license header and adds back the test-only inject constructor. Please review, [~yonigo]. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_v5.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: (was: OMID-117_v5.patch) > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_v5.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch, OMID-117_v5.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-90) Reducing begin/commit latency by distributing the write to the commit table
[ https://issues.apache.org/jira/browse/OMID-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646643#comment-16646643 ] James Taylor commented on OMID-90: -- [~ohads] - which commit? I'm not sure what you're asking. > Reducing begin/commit latency by distributing the write to the commit table > --- > > Key: OMID-90 > URL: https://issues.apache.org/jira/browse/OMID-90 > Project: Apache Omid > Issue Type: Sub-task >Reporter: Ohad Shacham >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OmidCloud-VLDB.pdf, omid90.patch > > > Today, Omid's commits are done by the transaction manager. In order to > efficiently write to the commit table, the transaction manager batches these > writes. This optimization, even thought reduces the write time to HBase, > significantly increases the begin and commit latency. The commit latency > increases since a commit operation returns only after its commit timestamp > was persisted in the commit table. And the begin latency increases since > begin returns a transaction id that is also used by the transaction to > identify its snapshot and therefore, begin returns only after all commits > with commit id smaller than the begin id was persisted in the commit table. > This is crucial, since a snapshot change during a transaction run may violate > snapshot isolation. > > The idea of this feature is to distribute the commit by moving the write to > the commit table from the server to the client. The transaction manager does > conflict analysis and returns a commit timestamp. While the client atomically > persists this commit in the commit table. > This significantly reduces the begin and commit latency, since batching is > not required anymore. A begin operation can immediately returns and a commit > operation returns after conflict detection. > This can introduce snapshot isolation violation since a slow client can > commit and change other transaction's snapsho. Therefore, we use an > invalidation technique which is similar to the one Omid uses today to > maintain snapshot isolation in high availability mode. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643998#comment-16643998 ] James Taylor edited comment on OMID-117 at 10/10/18 12:05 AM: -- As part of OMID-109 we'll need to move the coprocessor implementations into the shim module since Optional is not supported in Java 1.7 (which needs to be the target for hbase-1). We should try to make these implementations as thin as possible to reduce code duplication. We should be able to delegate to classes in hbase-common that do most of the work. I'll add the header where it's missing and add back the HBaseCommitTable constructor with the @Inject, but this should only be used for testing. I'll confirm that the hbase-1 unit tests pass with JDK 1.7, but we'll need to wait until OMID-109 is in place before being able to run the hbase-2 unit tests. was (Author: jamestaylor): As part of OMID-109 we'll need to move the coprocessor implementations into the shim module since Optional is not supported in Java 1.7 (which needs to be the target for hbase-1). We should try to make these implementations as thin as possible to reduce code duplication. We should be able to delegate to classes in hbase-common that do most of the work. I'll add the header where it's missing and add back the HBaseCommitTable constructor with the @Inject, but this should only be used for testing. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643998#comment-16643998 ] James Taylor edited comment on OMID-117 at 10/9/18 7:43 PM: As part of OMID-109 we'll need to move the coprocessor implementations into the shim module since Optional is not supported in Java 1.7 (which needs to be the target for hbase-1). We should try to make these implementations as thin as possible to reduce code duplication. We should be able to delegate to classes in hbase-common that do most of the work. I'll add the header where it's missing and add back the HBaseCommitTable constructor with the @Inject, but this should only be used for testing. was (Author: jamestaylor): As part of -OMID-1+09+-, we'll need to move the coprocessor implementations into the shim module since Optional is not supported in Java 1.7 (which needs to be the target for hbase-1). We should try to make these implementations as thin as possible to reduce code duplication. We should be able to delegate to classes in hbase-common that do most of the work. I'll add the header where it's missing and add back the HBaseCommitTable constructor with the @Inject, but this should only be used for testing. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643998#comment-16643998 ] James Taylor edited comment on OMID-117 at 10/9/18 7:42 PM: As part of -OMID-1+09+-, we'll need to move the coprocessor implementations into the shim module since Optional is not supported in Java 1.7 (which needs to be the target for hbase-1). We should try to make these implementations as thin as possible to reduce code duplication. We should be able to delegate to classes in hbase-common that do most of the work. I'll add the header where it's missing and add back the HBaseCommitTable constructor with the @Inject, but this should only be used for testing. was (Author: jamestaylor): As part of OMID-113, we'll need to move the coprocessor implementations into the shim module since Optional is not supported in Java 1.7 (which needs to be the target for hbase-1). We should try to make these implementations as thin as possible to reduce code duplication. We should be able to delegate to classes in hbase-common that do most of the work. I'll add the header where it's missing and add back the HBaseCommitTable constructor with the @Inject, but this should only be used for testing. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643998#comment-16643998 ] James Taylor commented on OMID-117: --- As part of OMID-113, we'll need to move the coprocessor implementations into the shim module since Optional is not supported in Java 1.7 (which needs to be the target for hbase-1). We should try to make these implementations as thin as possible to reduce code duplication. We should be able to delegate to classes in hbase-common that do most of the work. I'll add the header where it's missing and add back the HBaseCommitTable constructor with the @Inject, but this should only be used for testing. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641157#comment-16641157 ] James Taylor commented on OMID-117: --- For some reason git apply was ok, but git am was giving an error. I've removed the changes to ScrambledZipfianGenerator.java (which was just for an unnecessary cast) and attached a v4 which seems to work fine with git am as well. Please try again. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_v4.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch, > OMID-117_v4.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640930#comment-16640930 ] James Taylor commented on OMID-109: --- How about if we have separate compat modules for hbase-1 and hbase-2? These wouldn't be included in the top level pom. Instead, a pom that wants to include omid (like phoenix) would include two dependencies: the main omid one and the appropriate compat module. That's what tephra has done and it works well. You could do the same for the site building - make a separate module that's not included in the top level pom. WDYT, [~yonigo] and [~ohads]? > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_v3.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch, OMID-117_v3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640782#comment-16640782 ] James Taylor commented on OMID-117: --- Never mind. Turns out it was an environmental issue which went away after I rebooted. Please review the patch, [~ohads] or [~yonigo]. We still need to ensure that Java 1.7 is used for hbase-1 and Java 1.8 is used for hbase-2. I don't know how to do that with profiles, but if we use a compat module approach (similar to Tephra), it's pretty straightforward. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: (was: OMID-117_v2.patch) > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_v2.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-117: -- Attachment: OMID-117_v2.patch > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch, OMID-117_v2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640592#comment-16640592 ] James Taylor commented on OMID-117: --- Actually, looks like TestHBaseCommitTable hasn't passed since 81672f016b535546444ea1e3b551ae5dca4bf3ef. Any ideas? > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-117) Ensure timeouts are configured low for RPCs to commit table
[ https://issues.apache.org/jira/browse/OMID-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640577#comment-16640577 ] James Taylor commented on OMID-117: --- Need some help on this, [~ohads] and [~yonigo]. This patch adds new constructors to HBaseCommitTable so that you can pass the correctly configured Connection on the server side. However, I can't get TestHBaseCommitTable to pass. I'm not familiar with TestNG or Google Guice. Maybe I broke something? The strange this is that even if I try running the unit tests with an old commit, it still doesn't work. The key things on the server-side connection: * A single shared connection (rather than a new one created potentially per region) needs to be used, otherwise your region server will die when many regions attempt to connect to the commit table but are unable. * The timeouts need to be overridden because HBase by default multiplies the default settings by 10x for server to server RPCs (which would cause all you handler threads to get tied up and make your region server inaccessible. > Ensure timeouts are configured low for RPCs to commit table > --- > > Key: OMID-117 > URL: https://issues.apache.org/jira/browse/OMID-117 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-117.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-116) Add addShadowCells argument to other TTable methods
[ https://issues.apache.org/jira/browse/OMID-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636252#comment-16636252 ] James Taylor commented on OMID-116: --- Please review, [~yonigo] or [~ohads]. > Add addShadowCells argument to other TTable methods > --- > > Key: OMID-116 > URL: https://issues.apache.org/jira/browse/OMID-116 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-116.patch > > > We need to pass through the addShadowCells are to all the TTable methods that > can do puts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-116) Add addShadowCells argument to other TTable methods
[ https://issues.apache.org/jira/browse/OMID-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-116: -- Attachment: OMID-116.patch > Add addShadowCells argument to other TTable methods > --- > > Key: OMID-116 > URL: https://issues.apache.org/jira/browse/OMID-116 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-116.patch > > > We need to pass through the addShadowCells are to all the TTable methods that > can do puts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OMID-116) Add addShadowCells argument to other TTable methods
James Taylor created OMID-116: - Summary: Add addShadowCells argument to other TTable methods Key: OMID-116 URL: https://issues.apache.org/jira/browse/OMID-116 Project: Apache Omid Issue Type: Bug Reporter: James Taylor We need to pass through the addShadowCells are to all the TTable methods that can do puts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-115) Remove unnecessary throw of IOException from TTable.markPutAsCommitted()
[ https://issues.apache.org/jira/browse/OMID-115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-115: -- Attachment: OMID-115.patch > Remove unnecessary throw of IOException from TTable.markPutAsCommitted() > > > Key: OMID-115 > URL: https://issues.apache.org/jira/browse/OMID-115 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-115.patch > > > An IOException cannot possibly be thrown by TTable.markPutAsCommitted() as > the Cell that is added to the Put is the same one that was there before (and > hence the row key will always match and an IOException can never be thrown). > Better to ignore at the source than propagate it to the callers to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-115) Remove unnecessary throw of IOException from TTable.markPutAsCommitted()
[ https://issues.apache.org/jira/browse/OMID-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636177#comment-16636177 ] James Taylor commented on OMID-115: --- Simple patch. Please review, [~yonigo] or [~ohads]. > Remove unnecessary throw of IOException from TTable.markPutAsCommitted() > > > Key: OMID-115 > URL: https://issues.apache.org/jira/browse/OMID-115 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-115.patch > > > An IOException cannot possibly be thrown by TTable.markPutAsCommitted() as > the Cell that is added to the Put is the same one that was there before (and > hence the row key will always match and an IOException can never be thrown). > Better to ignore at the source than propagate it to the callers to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OMID-115) Remove unnecessary throw of IOException from TTable.markPutAsCommitted()
James Taylor created OMID-115: - Summary: Remove unnecessary throw of IOException from TTable.markPutAsCommitted() Key: OMID-115 URL: https://issues.apache.org/jira/browse/OMID-115 Project: Apache Omid Issue Type: Bug Reporter: James Taylor An IOException cannot possibly be thrown by TTable.markPutAsCommitted() as the Cell that is added to the Put is the same one that was there before (and hence the row key will always match and an IOException can never be thrown). Better to ignore at the source than propagate it to the callers to deal with. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (OMID-114) Prevent extra RPCs on TTable batch operations
[ https://issues.apache.org/jira/browse/OMID-114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor resolved OMID-114. --- Resolution: Fixed > Prevent extra RPCs on TTable batch operations > - > > Key: OMID-114 > URL: https://issues.apache.org/jira/browse/OMID-114 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-114.patch, OMID-114_v2.patch > > > We're doing an RPC for every item in batch operations on these TTable method: > * put(Transaction transaction, List puts) > * delete(Transaction transaction, List deletes) > * batch(Transaction transaction, List rows) > Instead, we can do a single RPC when the rows are sent to HBase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-114) Prevent extra RPCs on TTable batch operations
[
https://issues.apache.org/jira/browse/OMID-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635682#comment-16635682
]
James Taylor commented on OMID-114:
---
{quote}Why in the case of deletes we check deleteP.isEmpty() but we dont do it
for puts?
{quote}
I wanted to keep the same checks that were in place before my change. For
delete, there's a possibility for the Put to not contain anything - in the case
that the entire row is deleted and the Get doesn't find the row. Phoenix would
never hit this case as it always does a family delete. For the puts case, the
put can never be empty.
> Prevent extra RPCs on TTable batch operations
> -
>
> Key: OMID-114
> URL: https://issues.apache.org/jira/browse/OMID-114
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Priority: Major
> Attachments: OMID-114.patch, OMID-114_v2.patch
>
>
> We're doing an RPC for every item in batch operations on these TTable method:
> * put(Transaction transaction, List puts)
> * delete(Transaction transaction, List deletes)
> * batch(Transaction transaction, List rows)
> Instead, we can do a single RPC when the rows are sent to HBase.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor resolved OMID-113. --- Resolution: Fixed > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113_v2.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634793#comment-16634793 ] James Taylor commented on OMID-113: --- I think it's ok as is. I wouldn't want to call HTableDescriptor.hasCoprocessor in the TTable constructor as it'd add an extra RPC to get the HTableDescriptor. I think if you want to support native HBase clients, you should require them to add the coprocessor to their table (as Phoenix does). Then you can get rid of the boolean serverSideFilter completely. If you don't require the coprocessor, then if the native HBase client passes filters with their scans, it'll work incorrectly (just as it does with Phoenix). > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113_v2.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-113: -- Attachment: (was: OMID-113.patch) > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113_v2.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-113: -- Attachment: OMID-113_v2.patch > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch, OMID-113_v2.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-114) Prevent extra RPCs on TTable batch operations
[ https://issues.apache.org/jira/browse/OMID-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633414#comment-16633414 ] James Taylor commented on OMID-114: --- Ignore the first patch and look at v2 which corrects a few problems (based on running the TransactionIT unit test in Phoenix). > Prevent extra RPCs on TTable batch operations > - > > Key: OMID-114 > URL: https://issues.apache.org/jira/browse/OMID-114 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-114.patch, OMID-114_v2.patch > > > We're doing an RPC for every item in batch operations on these TTable method: > * put(Transaction transaction, List puts) > * delete(Transaction transaction, List deletes) > * batch(Transaction transaction, List rows) > Instead, we can do a single RPC when the rows are sent to HBase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-113) Set conflict free option when constructing TTable
[
https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633095#comment-16633095
]
James Taylor commented on OMID-113:
---
Question for you, [~yonigo] or [~ohads]: Why is there an option for
serverSideFilter in the constructor? Can we just get rid of that and always use
your new SnapshotFilterImpl? If not, should Phoenix pass true or false? Looks
like a value of false to this constructor causes SnapshotFilterImpl to be used
(which is a bit misleading based on the name of the parameter, no?):
{code:java}
public TTable(Table hTable, boolean serverSideFilter) throws IOException {
table = hTable;
mutations = new ArrayList();
this.serverSideFilter = serverSideFilter;
snapshotFilter = (serverSideFilter) ? new
AttributeSetSnapshotFilter(hTable) :
new SnapshotFilterImpl(new HTableAccessWrapper(hTable, hTable));
}
{code}
> Set conflict free option when constructing TTable
> -
>
> Key: OMID-113
> URL: https://issues.apache.org/jira/browse/OMID-113
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Priority: Major
> Attachments: OMID-113.patch
>
>
> Both Delete and Put mutations need to be able to be marked as conflict free.
> There no need to have an attribute store this information as it's only used
> on the client to determine which list in Transaction to put the mutation in.
> By having it as an attribute, you're only bloating the payload of the RPC.
> Instead, we can pass through in the TTable constructor whether it's a
> conflict free table or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-109: -- Priority: Blocker (was: Major) > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Blocker > Attachments: OMID-109.patch, omid109.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633053#comment-16633053 ] James Taylor commented on OMID-109: --- I tried a bunch of things to try to get Omid to build (none of which worked), but I'm unfortunately not very knowledgable about this. I'm getting an error in the Jenkins job that builds against the omid2 branch in Phoenix here that Java 1.8 is not a valid release target. See https://builds.apache.org/job/Phoenix-omid2/108/console > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OMID-109.patch, omid109.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633036#comment-16633036 ] James Taylor commented on OMID-113: --- You need to apply it on top of OMID-114, [~yonigo]. > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-113: -- Attachment: OMID-113.patch > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-113: -- Attachment: (was: OMID-113.patch) > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-113: -- Attachment: (was: OMID-113.patch) > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-113: -- Attachment: OMID-113.patch > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-113: -- Attachment: (was: OMID-113.patch) > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-113) Set conflict free option when constructing TTable
[ https://issues.apache.org/jira/browse/OMID-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632119#comment-16632119 ] James Taylor commented on OMID-113: --- Please review, [~ohads] or [~yonigo]. > Set conflict free option when constructing TTable > - > > Key: OMID-113 > URL: https://issues.apache.org/jira/browse/OMID-113 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-113.patch > > > Both Delete and Put mutations need to be able to be marked as conflict free. > There no need to have an attribute store this information as it's only used > on the client to determine which list in Transaction to put the mutation in. > By having it as an attribute, you're only bloating the payload of the RPC. > Instead, we can pass through in the TTable constructor whether it's a > conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-114) Prevent extra RPCs on TTable batch operations
[ https://issues.apache.org/jira/browse/OMID-114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632070#comment-16632070 ] James Taylor commented on OMID-114: --- Please review [~yonigo] or [~ohads]. > Prevent extra RPCs on TTable batch operations > - > > Key: OMID-114 > URL: https://issues.apache.org/jira/browse/OMID-114 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-114.patch > > > We're doing an RPC for every item in batch operations on these TTable method: > * put(Transaction transaction, List puts) > * delete(Transaction transaction, List deletes) > * batch(Transaction transaction, List rows) > Instead, we can do a single RPC when the rows are sent to HBase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OMID-114) Prevent extra RPCs on TTable batch operations
James Taylor created OMID-114: - Summary: Prevent extra RPCs on TTable batch operations Key: OMID-114 URL: https://issues.apache.org/jira/browse/OMID-114 Project: Apache Omid Issue Type: Sub-task Reporter: James Taylor We're doing an RPC for every item in batch operations on these TTable method: * put(Transaction transaction, List puts) * delete(Transaction transaction, List deletes) * batch(Transaction transaction, List rows) Instead, we can do a single RPC when the rows are sent to HBase. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (OMID-112) Make batch method signature compatible with Table.batch signature
[ https://issues.apache.org/jira/browse/OMID-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor closed OMID-112. - Resolution: Fixed > Make batch method signature compatible with Table.batch signature > - > > Key: OMID-112 > URL: https://issues.apache.org/jira/browse/OMID-112 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-112.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-112) Make batch method signature compatible with Table.batch signature
[ https://issues.apache.org/jira/browse/OMID-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632022#comment-16632022 ] James Taylor commented on OMID-112: --- Thanks for the review, [~yonigo]. I've fixed the javadoc and committed the change. > Make batch method signature compatible with Table.batch signature > - > > Key: OMID-112 > URL: https://issues.apache.org/jira/browse/OMID-112 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-112.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OMID-113) Set conflict free option when constructing TTable
James Taylor created OMID-113: - Summary: Set conflict free option when constructing TTable Key: OMID-113 URL: https://issues.apache.org/jira/browse/OMID-113 Project: Apache Omid Issue Type: Sub-task Reporter: James Taylor Both Delete and Put mutations need to be able to be marked as conflict free. There no need to have an attribute store this information as it's only used on the client to determine which list in Transaction to put the mutation in. By having it as an attribute, you're only bloating the payload of the RPC. Instead, we can pass through in the TTable constructor whether it's a conflict free table or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-112) Make batch method signature compatible with Table.batch signature
[ https://issues.apache.org/jira/browse/OMID-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-112: -- Attachment: OMID-112.patch > Make batch method signature compatible with Table.batch signature > - > > Key: OMID-112 > URL: https://issues.apache.org/jira/browse/OMID-112 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-112.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-112) Make batch method signature compatible with Table.batch signature
[ https://issues.apache.org/jira/browse/OMID-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-112: -- Attachment: (was: OMID-82.patch) > Make batch method signature compatible with Table.batch signature > - > > Key: OMID-112 > URL: https://issues.apache.org/jira/browse/OMID-112 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-112) Make batch method signature compatible with Table.batch signature
[ https://issues.apache.org/jira/browse/OMID-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630704#comment-16630704 ] James Taylor commented on OMID-112: --- Please review, [~ohads] or [~ygor...@yahoo.com]. > Make batch method signature compatible with Table.batch signature > - > > Key: OMID-112 > URL: https://issues.apache.org/jira/browse/OMID-112 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-82.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OMID-112) Make batch method signature compatible with Table.batch signature
[ https://issues.apache.org/jira/browse/OMID-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16630704#comment-16630704 ] James Taylor edited comment on OMID-112 at 9/27/18 4:32 PM: Please review, [~ohads] or [~yonigo]. was (Author: jamestaylor): Please review, [~ohads] or [~ygor...@yahoo.com]. > Make batch method signature compatible with Table.batch signature > - > > Key: OMID-112 > URL: https://issues.apache.org/jira/browse/OMID-112 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-82.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-112) Make batch method signature compatible with Table.batch signature
[ https://issues.apache.org/jira/browse/OMID-112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-112: -- Attachment: OMID-82.patch > Make batch method signature compatible with Table.batch signature > - > > Key: OMID-112 > URL: https://issues.apache.org/jira/browse/OMID-112 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-82.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (OMID-82) Integrating Omid with Phoenix
[ https://issues.apache.org/jira/browse/OMID-82?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-82: - Attachment: OMID-82.patch > Integrating Omid with Phoenix > - > > Key: OMID-82 > URL: https://issues.apache.org/jira/browse/OMID-82 > Project: Apache Omid > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Ohad Shacham >Priority: Major > Attachments: OMID-82.patch > > > This Jira encapsulates required augmentations to Omid in order to integrate > with Apache Phoenix. [PHOENIX-3623] contains the integration workplace, as > well as the required augmentations and modification to Apache Phoenix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (OMID-112) Make batch method signature compatible with Table.batch signature
James Taylor created OMID-112: - Summary: Make batch method signature compatible with Table.batch signature Key: OMID-112 URL: https://issues.apache.org/jira/browse/OMID-112 Project: Apache Omid Issue Type: Sub-task Reporter: James Taylor -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16617684#comment-16617684 ] James Taylor commented on OMID-109: --- Will do. Please let me know once it’s been committed. > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Assignee: Yonatan Gottesman >Priority: Major > Attachments: OMID-109.patch, omid109.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[ https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613641#comment-16613641 ] James Taylor commented on OMID-107: --- WDYT, [~ohads]? > Replace pre 1.0 deprecated HBase APIs > - > > Key: OMID-107 > URL: https://issues.apache.org/jira/browse/OMID-107 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-107.patch, OMID-107_v2.patch > > > In HBase 2.0, the HTableInterface has been replaced with the Table interface. > Thus, in Omid, this same replacement should take place. This will mostly be > isolated to having TTable implement Table instead of HTableInterface. This > will be required to check-in Omid support in Phoenix, since Phoenix 5.x is > based on HBase 2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[ https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612219#comment-16612219 ] James Taylor edited comment on OMID-107 at 9/12/18 5:59 PM: Neither Phoenix or HBase will do further 0.98 releases. Phoenix won’t have a 0.98 release for Omid support. Why not have a separate legacy branch for critical 0.98 bug fixes since the current changes to Omid aren’t needed as they are all related to Phoenix integration? Otherwise you’ll have to pollute your API by going through the shim layer for adding columns to Put/Delete, a wrapper for HTableInterface, a wrapper for HBaseAdmin, etc. It’ll be pretty ugly and cumbersome. was (Author: jamestaylor): Neither Phoenix or HBase will do further 0.98 releases. Phoenix won’t have a 0.98 release for Omid support. Why not have a separate legacy branch for critical 0.98 bug fixes as the current changes to Omid aren’t needed apart from integrating with Phoenix? Otherwise you’ll have to pollute your API by going through the shim layer for adding columns to Put/Delete, a wrapper for HTableInterface, a wrapper for Admin, etc. It’ll be pretty ugly and cumbersome. > Replace pre 1.0 deprecated HBase APIs > - > > Key: OMID-107 > URL: https://issues.apache.org/jira/browse/OMID-107 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-107.patch, OMID-107_v2.patch > > > In HBase 2.0, the HTableInterface has been replaced with the Table interface. > Thus, in Omid, this same replacement should take place. This will mostly be > isolated to having TTable implement Table instead of HTableInterface. This > will be required to check-in Omid support in Phoenix, since Phoenix 5.x is > based on HBase 2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[ https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612219#comment-16612219 ] James Taylor commented on OMID-107: --- Neither Phoenix or HBase will do further 0.98 releases. Phoenix won’t have a 0.98 release for Omid support. Why not have a separate legacy branch for critical 0.98 bug fixes as the current changes to Omid aren’t needed apart from integrating with Phoenix? Otherwise you’ll have to pollute your API by going through the shim layer for adding columns to Put/Delete, a wrapper for HTableInterface, a wrapper for Admin, etc. It’ll be pretty ugly and cumbersome. > Replace pre 1.0 deprecated HBase APIs > - > > Key: OMID-107 > URL: https://issues.apache.org/jira/browse/OMID-107 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-107.patch, OMID-107_v2.patch > > > In HBase 2.0, the HTableInterface has been replaced with the Table interface. > Thus, in Omid, this same replacement should take place. This will mostly be > isolated to having TTable implement Table instead of HTableInterface. This > will be required to check-in Omid support in Phoenix, since Phoenix 5.x is > based on HBase 2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[ https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603442#comment-16603442 ] James Taylor commented on OMID-109: --- Thanks, [~mujtabachohan]. When I try to navigate to that link, I get an Access Denied exception: jamestaylor is missing the Job/ExtendedRead permission. Any chance you could make that change? Or if the effort is large, then no problem. We just want to get confirmation that all the Phoenix tests pass now - I've run all the failing ones locally and they all pass now. > Unable to build phoenix-integration branch through Jenkins job > -- > > Key: OMID-109 > URL: https://issues.apache.org/jira/browse/OMID-109 > Project: Apache Omid > Issue Type: Bug >Reporter: James Taylor >Priority: Major > Attachments: OMID-109.patch > > > Based on Jenkins job failures > (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom > needs to be updated to > https://raw.githubusercontent.com/synergian/wagon-git/releases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-110) Support HBase 2.0
[
https://issues.apache.org/jira/browse/OMID-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603387#comment-16603387
]
James Taylor commented on OMID-110:
---
[~yonigo] - any progress on this one? FWIW, the way Tephra does compat modules
works pretty well. They have the HBase dependencies specified in their shim
module. Other projects that want to use Tephra specify two dependencies: the
tephra core dependency and the tephra compat module (based on the version of
HBase they use).
> Support HBase 2.0
> -
>
> Key: OMID-110
> URL: https://issues.apache.org/jira/browse/OMID-110
> Project: Apache Omid
> Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OMID-110_wip1.patch
>
>
> Once OMID-107 is committed (which replaced all deprecated methods), the only
> things I've identified that need to change are:
> * Instantiation of CellComparator should be done through HBaseShims since
> it's different between HBase 1.x and 2.x. In HBase 2.0, CellComparator was
> changed to an interface. Another alternative would be to write our own
> CellComparator.
> {code:java}
> public static SortedMap>
> mapCellsToShadowCells(List cells) {
> // Move CellComparator to HBaseSims for 2.0 support
> // Need to access through CellComparatorImpl.COMPARATOR
> SortedMap> cellToShadowCellMap
> = new TreeMap>(new CellComparator());{code}
> The HBase 2.0 version would use CellComparatorImpl.COMPARATOR instead of
> instantiating a CellComparator.
> * Add createTable method to HBaseShims. Instead of HTableDescriptor, HBase
> 2.0 uses TableDescriptor and TableDescriptorBuilder.
> [~chrajeshbab...@gmail.com] and [~an...@apache.org] - do you know why
> existing admin.createTable(HTableDescriptor) wouldn't work? HTableDescriptor
> implements TableDescriptor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OMID-110) Support HBase 2.0
[
https://issues.apache.org/jira/browse/OMID-110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Taylor updated OMID-110:
--
Summary: Support HBase 2.0 (was: Create an hbase-2 profile to support
HBase 2.0 release)
> Support HBase 2.0
> -
>
> Key: OMID-110
> URL: https://issues.apache.org/jira/browse/OMID-110
> Project: Apache Omid
> Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OMID-110_wip1.patch
>
>
> Once OMID-107 is committed (which replaced all deprecated methods), the only
> things I've identified that need to change are:
> * Instantiation of CellComparator should be done through HBaseShims since
> it's different between HBase 1.x and 2.x. In HBase 2.0, CellComparator was
> changed to an interface. Another alternative would be to write our own
> CellComparator.
> {code:java}
> public static SortedMap>
> mapCellsToShadowCells(List cells) {
> // Move CellComparator to HBaseSims for 2.0 support
> // Need to access through CellComparatorImpl.COMPARATOR
> SortedMap> cellToShadowCellMap
> = new TreeMap>(new CellComparator());{code}
> The HBase 2.0 version would use CellComparatorImpl.COMPARATOR instead of
> instantiating a CellComparator.
> * Add createTable method to HBaseShims. Instead of HTableDescriptor, HBase
> 2.0 uses TableDescriptor and TableDescriptorBuilder.
> [~chrajeshbab...@gmail.com] and [~an...@apache.org] - do you know why
> existing admin.createTable(HTableDescriptor) wouldn't work? HTableDescriptor
> implements TableDescriptor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OMID-110) Create an hbase-2 profile to support HBase 2.0 release
[
https://issues.apache.org/jira/browse/OMID-110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Taylor updated OMID-110:
--
Attachment: OMID-110_wip1.patch
> Create an hbase-2 profile to support HBase 2.0 release
> --
>
> Key: OMID-110
> URL: https://issues.apache.org/jira/browse/OMID-110
> Project: Apache Omid
> Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Major
> Attachments: OMID-110_wip1.patch
>
>
> Once OMID-107 is committed (which replaced all deprecated methods), the only
> things I've identified that need to change are:
> * Instantiation of CellComparator should be done through HBaseShims since
> it's different between HBase 1.x and 2.x. In HBase 2.0, CellComparator was
> changed to an interface. Another alternative would be to write our own
> CellComparator.
> {code:java}
> public static SortedMap>
> mapCellsToShadowCells(List cells) {
> // Move CellComparator to HBaseSims for 2.0 support
> // Need to access through CellComparatorImpl.COMPARATOR
> SortedMap> cellToShadowCellMap
> = new TreeMap>(new CellComparator());{code}
> The HBase 2.0 version would use CellComparatorImpl.COMPARATOR instead of
> instantiating a CellComparator.
> * Add createTable method to HBaseShims. Instead of HTableDescriptor, HBase
> 2.0 uses TableDescriptor and TableDescriptorBuilder.
> [~chrajeshbab...@gmail.com] and [~an...@apache.org] - do you know why
> existing admin.createTable(HTableDescriptor) wouldn't work? HTableDescriptor
> implements TableDescriptor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[ https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595102#comment-16595102 ] James Taylor commented on OMID-107: --- +1. Agree that those should be removed. > Replace pre 1.0 deprecated HBase APIs > - > > Key: OMID-107 > URL: https://issues.apache.org/jira/browse/OMID-107 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-107.patch, OMID-107_v2.patch > > > In HBase 2.0, the HTableInterface has been replaced with the Table interface. > Thus, in Omid, this same replacement should take place. This will mostly be > isolated to having TTable implement Table instead of HTableInterface. This > will be required to check-in Omid support in Phoenix, since Phoenix 5.x is > based on HBase 2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OMID-110) Create an hbase-2 profile to support HBase 2.0 release
[
https://issues.apache.org/jira/browse/OMID-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593227#comment-16593227
]
James Taylor commented on OMID-110:
---
Thanks, [~ankit.singhal]. Must be related to the dependencies not being right
(guava, zk, hadoop, etc). Not sure of the best way for Omid to manage that. I
think they may need to move some dependencies into the shim module. Any advice?
> Create an hbase-2 profile to support HBase 2.0 release
> --
>
> Key: OMID-110
> URL: https://issues.apache.org/jira/browse/OMID-110
> Project: Apache Omid
> Issue Type: Improvement
>Reporter: James Taylor
>Assignee: Yonatan Gottesman
>Priority: Major
>
> Once OMID-107 is committed (which replaced all deprecated methods), the only
> things I've identified that need to change are:
> * Instantiation of CellComparator should be done through HBaseShims since
> it's different between HBase 1.x and 2.x. In HBase 2.0, CellComparator was
> changed to an interface. Another alternative would be to write our own
> CellComparator.
> {code:java}
> public static SortedMap>
> mapCellsToShadowCells(List cells) {
> // Move CellComparator to HBaseSims for 2.0 support
> // Need to access through CellComparatorImpl.COMPARATOR
> SortedMap> cellToShadowCellMap
> = new TreeMap>(new CellComparator());{code}
> The HBase 2.0 version would use CellComparatorImpl.COMPARATOR instead of
> instantiating a CellComparator.
> * Add createTable method to HBaseShims. Instead of HTableDescriptor, HBase
> 2.0 uses TableDescriptor and TableDescriptorBuilder.
> [~chrajeshbab...@gmail.com] and [~an...@apache.org] - do you know why
> existing admin.createTable(HTableDescriptor) wouldn't work? HTableDescriptor
> implements TableDescriptor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[
https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592818#comment-16592818
]
James Taylor edited comment on OMID-107 at 8/26/18 5:21 PM:
Thanks for the debugging, [~yonigo]. All the tests pass with v2 now. I've
updated the pom to HBase 1.2.5 (min HBase version supported by Phoenix) and
confirmed that all tests pass with both HBase 1.2, HBase 1.3 and HBase 1.4
(currently supported HBase 1.x versions of Phoenix). I also tried with HBase
2.0 (used by current master of Phoenix) and there are only a few things that
need to be added to the shim layer:
* Instantiation of CellComparator. In HBase 2.0, CellComparator was changed to
an interface. Perhaps we can write our own CellComparator that's simpler?
{code:java}
public static SortedMap> mapCellsToShadowCells(List
cells) {
// Move CellComparator to HBaseSims for 2.0 support
// Need to access through CellComparatorImpl.COMPARATOR
SortedMap> cellToShadowCellMap
= new TreeMap>(new CellComparator());{code}
* Code that creates tables through Admin. Instead of HTableDescriptor,
TableDescriptor and TableDescriptorBuilder need to be used.
I've filed OMID-110 for creating an hbase2 profile.
FYI, [~ohads]. Please review.
was (Author: jamestaylor):
Thanks for the debugging, [~yonigo]. All the tests pass with v2 now. I've
updated the pom to HBase 1.3.1 and confirmed that all tests pass with both
HBase 1.2, HBase 1.3 and HBase 1.4 (currently supported HBase 1.x versions of
Phoenix). I also tried with HBase 2.0 (used by current master of Phoenix) and
there are only a few things that need to be added to the shim layer:
* Instantiation of CellComparator. In HBase 2.0, CellComparator was changed to
an interface. Perhaps we can write our own CellComparator that's simpler?
{code:java}
public static SortedMap> mapCellsToShadowCells(List
cells) {
// Move CellComparator to HBaseSims for 2.0 support
// Need to access through CellComparatorImpl.COMPARATOR
SortedMap> cellToShadowCellMap
= new TreeMap>(new CellComparator());{code}
* Code that creates tables through Admin. Instead of HTableDescriptor,
TableDescriptor and TableDescriptorBuilder need to be used.
I've filed OMID-110 for creating an hbase2 profile.
FYI, [~ohads]. Please review.
> Replace pre 1.0 deprecated HBase APIs
> -
>
> Key: OMID-107
> URL: https://issues.apache.org/jira/browse/OMID-107
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Priority: Major
> Attachments: OMID-107.patch, OMID-107_v2.patch
>
>
> In HBase 2.0, the HTableInterface has been replaced with the Table interface.
> Thus, in Omid, this same replacement should take place. This will mostly be
> isolated to having TTable implement Table instead of HTableInterface. This
> will be required to check-in Omid support in Phoenix, since Phoenix 5.x is
> based on HBase 2.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[ https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-107: -- Attachment: OMID-107_v2.patch > Replace pre 1.0 deprecated HBase APIs > - > > Key: OMID-107 > URL: https://issues.apache.org/jira/browse/OMID-107 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-107.patch, OMID-107_v2.patch > > > In HBase 2.0, the HTableInterface has been replaced with the Table interface. > Thus, in Omid, this same replacement should take place. This will mostly be > isolated to having TTable implement Table instead of HTableInterface. This > will be required to check-in Omid support in Phoenix, since Phoenix 5.x is > based on HBase 2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[
https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592818#comment-16592818
]
James Taylor edited comment on OMID-107 at 8/26/18 5:13 PM:
Thanks for the debugging, [~yonigo]. All the tests pass with v2 now. I've
updated the pom to HBase 1.3.1 and confirmed that all tests pass with both
HBase 1.2, HBase 1.3 and HBase 1.4 (currently supported HBase 1.x versions of
Phoenix). I also tried with HBase 2.0 (used by current master of Phoenix) and
there are only a few things that need to be added to the shim layer:
* Instantiation of CellComparator. In HBase 2.0, CellComparator was changed to
an interface. Perhaps we can write our own CellComparator that's simpler?
{code:java}
public static SortedMap> mapCellsToShadowCells(List
cells) {
// Move CellComparator to HBaseSims for 2.0 support
// Need to access through CellComparatorImpl.COMPARATOR
SortedMap> cellToShadowCellMap
= new TreeMap>(new CellComparator());{code}
* Code that creates tables through Admin. Instead of HTableDescriptor,
TableDescriptor and TableDescriptorBuilder need to be used.
I've filed OMID-110 for creating an hbase2 profile.
FYI, [~ohads]. Please review.
was (Author: jamestaylor):
Thanks for the info, [~yonigo]. All the tests pass with v2. There are still a
few things left to do:
* Change pom to use HBase 1.3.0
* Ensure it works against HBase 1.4.0. I was seeing a compilation failure when
I tried it.
I also tried with HBase 2.0 and there are only a few things that need to be
added to the shim layer:
* Instantiation of CellComparator. In HBase 2.0, CellComparator was changed to
an interface. Perhaps we can write our own CellComparator that's simpler?
{code:java}
public static SortedMap> mapCellsToShadowCells(List
cells) {
// Move CellComparator to HBaseSims for 2.0 support
// Need to access through CellComparatorImpl.COMPARATOR
SortedMap> cellToShadowCellMap
= new TreeMap>(new CellComparator());{code}
* Code that creates tables through Admin. Instead of HTableDescriptor,
TableDescriptor and TableDescriptorBuilder need to be used.
FYI, [~ohads].
> Replace pre 1.0 deprecated HBase APIs
> -
>
> Key: OMID-107
> URL: https://issues.apache.org/jira/browse/OMID-107
> Project: Apache Omid
> Issue Type: Sub-task
>Reporter: James Taylor
>Priority: Major
> Attachments: OMID-107.patch, OMID-107_v2.patch
>
>
> In HBase 2.0, the HTableInterface has been replaced with the Table interface.
> Thus, in Omid, this same replacement should take place. This will mostly be
> isolated to having TTable implement Table instead of HTableInterface. This
> will be required to check-in Omid support in Phoenix, since Phoenix 5.x is
> based on HBase 2.0.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OMID-109) Unable to build phoenix-integration branch through Jenkins job
[
https://issues.apache.org/jira/browse/OMID-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16592962#comment-16592962
]
James Taylor commented on OMID-109:
---
[~mujtabachohan] - would a workaround be to add the command you mentioned above
before the mvn install?
{code:java}
mvn install:install-file -DgroupId=ar.com.synergian -DartifactId=wagon-git
-Dversion=0.2.5 -Dpackaging=jar -Dfile=/path/to/file
{code}
[~ohads], [~yonigo] - looks like the long term fix is to get
ar.com.synergian:wagon-git hosted on Maven.
> Unable to build phoenix-integration branch through Jenkins job
> --
>
> Key: OMID-109
> URL: https://issues.apache.org/jira/browse/OMID-109
> Project: Apache Omid
> Issue Type: Bug
>Reporter: James Taylor
>Priority: Major
> Attachments: OMID-109.patch
>
>
> Based on Jenkins job failures
> (https://builds.apache.org/job/Phoenix-omid2/81/), the repo URL in the pom
> needs to be updated to
> https://raw.githubusercontent.com/synergian/wagon-git/releases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (OMID-107) Replace pre 1.0 deprecated HBase APIs
[ https://issues.apache.org/jira/browse/OMID-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated OMID-107: -- Attachment: (was: OMID-107_v2.patch) > Replace pre 1.0 deprecated HBase APIs > - > > Key: OMID-107 > URL: https://issues.apache.org/jira/browse/OMID-107 > Project: Apache Omid > Issue Type: Sub-task >Reporter: James Taylor >Priority: Major > Attachments: OMID-107.patch, OMID-107_v2.patch > > > In HBase 2.0, the HTableInterface has been replaced with the Table interface. > Thus, in Omid, this same replacement should take place. This will mostly be > isolated to having TTable implement Table instead of HTableInterface. This > will be required to check-in Omid support in Phoenix, since Phoenix 5.x is > based on HBase 2.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
