[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516962#comment-16516962 ] James Taylor commented on PHOENIX-3623: --- This is almost ready to merge, [~ohads] & [~tdsilva]. There are three outstanding Omid issues that need to be fixed first which are causing nine test failures: * OMID-102 Implement visibility filter as pure HBase Filter ** FlappingTransactionIT.testInflightUpdateNotSeen[FlappingTransactionIT_transactionProvider=OMID] ** FlappingTransactionIT.testInflightDeleteNotSeen[FlappingTransactionIT_transactionProvider=OMID] * OMID-103 DeleteFamily marker still visible after rollback fails ** TransactionIT.testFailureToRollbackAfterDelete[TransactionIT_provider=OMID] *(added specifically to repro issue in a simple way)* ** MutableIndexFailureIT.testIndexWriteFailure[MutableIndexFailureIT_transactionProvider=OMID,localIndex=true,isNamespaceMapped=false,disableIndexOnWriteFailure=false,failRebuildTask=false,throwIndexWriteFailure=null] ** MutableIndexFailureIT.testIndexWriteFailure[MutableIndexFailureIT_transactionProvider=OMID,localIndex=true,isNamespaceMapped=true,disableIndexOnWriteFailure=false,failRebuildTask=false,throwIndexWriteFailure=null] ** MutableIndexFailureIT.testIndexWriteFailure[MutableIndexFailureIT_transactionProvider=OMID,localIndex=false,isNamespaceMapped=false,disableIndexOnWriteFailure=false,failRebuildTask=false,throwIndexWriteFailure=null] ** MutableIndexFailureIT.testIndexWriteFailure[MutableIndexFailureIT_transactionProvider=OMID,localIndex=false,isNamespaceMapped=true,disableIndexOnWriteFailure=false,failRebuildTask=false,throwIndexWriteFailure=null] * PHOENIX-4783 Fix Timestamp not allowed in transactional user operations error ** MutableRollbackIT.testCheckpointAndRollback[MutableRollbackIT_localIndex=false,transactionProvider=OMID] ** MutableRollbackIT.testCheckpointAndRollback[MutableRollbackIT_localIndex=true,transactionProvider=OMID] One minor and trivial to fix issue is (currently being worked around) is PHOENIX-4782 (Set transaction on OmidTransactionTable or throw if created called before transaction started) One important missing feature that would be useful to add: OMID-104 Support TTL > Integrate Omid with Phoenix > --- > > Key: PHOENIX-3623 > URL: https://issues.apache.org/jira/browse/PHOENIX-3623 > Project: Phoenix > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Ohad Shacham >Priority: Major > Fix For: 4.15.0 > > > The purpose of this Jira is to propose a work plan for connecting Omid to > Phoenix. > Each task of the following will be handled in a seperate sub Jira. Subtasks > 4.* are related to augmenting Omid to support features required by Phoenix > and therefore, their corresponding Jiras will appear under Omid and not under > Phoenix. > Each task is completed by a commit. > Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls > are integrated inside Phoenix code. Therefore, in order to support both Omid > and Tephra, we need to add another abstraction layer that later-on will be > connected to both Tephra and Omid. The first tasks is to define such an > interface. > Task 2: Implement TAL functionality for Tephra. > Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra. > Task 4: Implement Omid required features for Phoenix: > Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction > where every write occurs after the checkpoint is not visible by the > transaction. Explanations for this feature can be seen in [TEPHRA-96]. > Task 4.2: Add an option to mark a key as non-conflicting. The motivation is > to reduce the size of the write set needed by the transaction manager upon > commit as well as reduce the conflict detection work. > Task 4.3: Add support for transactions that never abort. Such transactions > will only make other inflight transactions abort and will abort only in case > of a transaction manager failure. > These transactions are needed for ‘create index’ and the scenario was > discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind > of transactions was also discussed in [OMID-56]. > Task 4.4: Add support for returning multiple versions in a scan. The use case > is described in [TEPHRA-134]. > Task 4.5: Change Omid's timestamp mechanism to return real time based > timestamp, while keeping monotonicity. > Task 5: Implement TAL functionality for Omid. > Task 6: Implement performance tests and tune Omid for Phoenix use. This task > requires understanding of common usage scenarios in Phoenix as well as > defining the tradeoff between throughput and latency. > Could you please review the proposed work plan? > Also, could you please let me know whether I missed any
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504749#comment-16504749
]
James Taylor commented on PHOENIX-3623:
---
Thanks, [~ohads]. You mentioned before that you knew how to fix this one, right?
{code}
ERROR] Tests run: 6, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 84.426 s
<<< FAILURE! - in org.apache.phoenix.end2end.index.txn.MutableRollbackIT
[ERROR]
testCheckpointAndRollback[MutableRollbackIT_localIndex=false](org.apache.phoenix.end2end.index.txn.MutableRollbackIT)
Time elapsed: 8.822 s <<< ERROR!
org.apache.phoenix.execute.CommitException: java.lang.IllegalArgumentException:
Timestamp not allowed in transactional user operations
at
org.apache.phoenix.end2end.index.txn.MutableRollbackIT.testCheckpointAndRollback(MutableRollbackIT.java:475)
Caused by: java.lang.IllegalArgumentException: Timestamp not allowed in
transactional user operations
at
org.apache.phoenix.end2end.index.txn.MutableRollbackIT.testCheckpointAndRollback(MutableRollbackIT.java:475)
[ERROR]
testCheckpointAndRollback[MutableRollbackIT_localIndex=true](org.apache.phoenix.end2end.index.txn.MutableRollbackIT)
Time elapsed: 9.919 s <<< ERROR!
org.apache.phoenix.execute.CommitException: java.lang.IllegalArgumentException:
Timestamp not allowed in transactional user operations
at
org.apache.phoenix.end2end.index.txn.MutableRollbackIT.testCheckpointAndRollback(MutableRollbackIT.java:475)
Caused by: java.lang.IllegalArgumentException: Timestamp not allowed in
transactional user operations
at
org.apache.phoenix.end2end.index.txn.MutableRollbackIT.testCheckpointAndRollback(MutableRollbackIT.java:475)
{code}
> Integrate Omid with Phoenix
> ---
>
> Key: PHOENIX-3623
> URL: https://issues.apache.org/jira/browse/PHOENIX-3623
> Project: Phoenix
> Issue Type: New Feature
>Reporter: Ohad Shacham
>Assignee: Ohad Shacham
>Priority: Major
>
> The purpose of this Jira is to propose a work plan for connecting Omid to
> Phoenix.
> Each task of the following will be handled in a seperate sub Jira. Subtasks
> 4.* are related to augmenting Omid to support features required by Phoenix
> and therefore, their corresponding Jiras will appear under Omid and not under
> Phoenix.
> Each task is completed by a commit.
> Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls
> are integrated inside Phoenix code. Therefore, in order to support both Omid
> and Tephra, we need to add another abstraction layer that later-on will be
> connected to both Tephra and Omid. The first tasks is to define such an
> interface.
> Task 2: Implement TAL functionality for Tephra.
> Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra.
> Task 4: Implement Omid required features for Phoenix:
> Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction
> where every write occurs after the checkpoint is not visible by the
> transaction. Explanations for this feature can be seen in [TEPHRA-96].
> Task 4.2: Add an option to mark a key as non-conflicting. The motivation is
> to reduce the size of the write set needed by the transaction manager upon
> commit as well as reduce the conflict detection work.
> Task 4.3: Add support for transactions that never abort. Such transactions
> will only make other inflight transactions abort and will abort only in case
> of a transaction manager failure.
> These transactions are needed for ‘create index’ and the scenario was
> discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind
> of transactions was also discussed in [OMID-56].
> Task 4.4: Add support for returning multiple versions in a scan. The use case
> is described in [TEPHRA-134].
> Task 4.5: Change Omid's timestamp mechanism to return real time based
> timestamp, while keeping monotonicity.
> Task 5: Implement TAL functionality for Omid.
> Task 6: Implement performance tests and tune Omid for Phoenix use. This task
> requires understanding of common usage scenarios in Phoenix as well as
> defining the tradeoff between throughput and latency.
> Could you please review the proposed work plan?
> Also, could you please let me know whether I missed any augmentation needed
> for Omid in order to support Phoenix operations?
> I opened a jira [OMID-82] that encapsulates all Omid related development for
> Phoenix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504736#comment-16504736 ] Ohad Shacham commented on PHOENIX-3623: --- [~jamestaylor], This is the output after committing the fences fix to Omid. nohup: ignoring input /home/y/libexec/maven/bin/mvn verify [INFO] Scanning for projects... [INFO] [INFO] Reactor Build Order: [INFO] [INFO] Apache Phoenix [INFO] Phoenix Core [INFO] Phoenix - Flume [INFO] Phoenix - Kafka [INFO] Phoenix - Pig [INFO] Phoenix Query Server Client [INFO] Phoenix Query Server [INFO] Phoenix - Pherf [INFO] Phoenix - Spark [INFO] Phoenix - Hive [INFO] Phoenix Client [INFO] Phoenix Server [INFO] Phoenix Assembly [INFO] Phoenix - Tracing Web Application [INFO] Phoenix Load Balancer [INFO] [INFO] [INFO] Building Apache Phoenix 4.14.0-HBase-1.3 [INFO] [INFO] [INFO] --- maven-checkstyle-plugin:2.13:check (validate) @ phoenix --- [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ phoenix --- [INFO] [INFO] --- maven-source-plugin:2.2.1:jar-no-fork (attach-sources) @ phoenix --- [INFO] [INFO] --- maven-jar-plugin:2.4:test-jar (default) @ phoenix --- [WARNING] JAR will be empty - no content was marked for inclusion! [INFO] [INFO] --- maven-site-plugin:3.2:attach-descriptor (attach-descriptor) @ phoenix --- [INFO] [INFO] [INFO] Building Phoenix Core 4.14.0-HBase-1.3 [INFO] Downloading: https://repository.apache.org/content/repositories/releases/org/apache/omid/omid-hbase-client/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://ymaven.corp.yahoo.com:/proximity/repository/apache.snapshot/org/apache/omid/omid-hbase-client/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://edge.artifactory.yahoo.com:8000/artifactory/maven-local-snapshot/org/apache/omid/omid-hbase-client/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://edge.artifactory.yahoo.com:8000/artifactory/maven-local-snapshot/org/apache/omid/omid/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/content/repositories/releases/org/apache/omid/omid/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://ymaven.corp.yahoo.com:/proximity/repository/apache.snapshot/org/apache/omid/omid/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://edge.artifactory.yahoo.com:8000/artifactory/maven-local-snapshot/org/apache/omid/omid-transaction-client/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/content/repositories/releases/org/apache/omid/omid-transaction-client/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://ymaven.corp.yahoo.com:/proximity/repository/apache.snapshot/org/apache/omid/omid-transaction-client/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://edge.artifactory.yahoo.com:8000/artifactory/maven-local-snapshot/org/apache/omid/omid-common/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/content/repositories/releases/org/apache/omid/omid-common/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://ymaven.corp.yahoo.com:/proximity/repository/apache.snapshot/org/apache/omid/omid-common/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://edge.artifactory.yahoo.com:8000/artifactory/maven-local-snapshot/org/apache/omid/omid-statemachine/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/content/repositories/releases/org/apache/omid/omid-statemachine/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://ymaven.corp.yahoo.com:/proximity/repository/apache.snapshot/org/apache/omid/omid-statemachine/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://edge.artifactory.yahoo.com:8000/artifactory/maven-local-snapshot/org/apache/omid/omid-commit-table/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/content/repositories/releases/org/apache/omid/omid-commit-table/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://ymaven.corp.yahoo.com:/proximity/repository/apache.snapshot/org/apache/omid/omid-commit-table/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://edge.artifactory.yahoo.com:8000/artifactory/maven-local-snapshot/org/apache/omid/omid-metrics/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: https://repository.apache.org/content/repositories/releases/org/apache/omid/omid-metrics/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading: http://ymaven.corp.yahoo.com:/proximity/repository/apache.snapshot/org/apache/omid/omid-metrics/0.8.2.11-SNAPSHOT/maven-metadata.xml Downloading:
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16496388#comment-16496388 ] Ohad Shacham commented on PHOENIX-3623: --- Weird, GlobalImmutableTxIndexIT.testCreateIndexAfterUpsertStarted used to pass. Fence is Omid cannot fail. Once a fence request occurs a fence will be made which guarantee that every inflight transaction that access the fence table during the transaction will be aborted. > Integrate Omid with Phoenix > --- > > Key: PHOENIX-3623 > URL: https://issues.apache.org/jira/browse/PHOENIX-3623 > Project: Phoenix > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Ohad Shacham >Priority: Major > > The purpose of this Jira is to propose a work plan for connecting Omid to > Phoenix. > Each task of the following will be handled in a seperate sub Jira. Subtasks > 4.* are related to augmenting Omid to support features required by Phoenix > and therefore, their corresponding Jiras will appear under Omid and not under > Phoenix. > Each task is completed by a commit. > Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls > are integrated inside Phoenix code. Therefore, in order to support both Omid > and Tephra, we need to add another abstraction layer that later-on will be > connected to both Tephra and Omid. The first tasks is to define such an > interface. > Task 2: Implement TAL functionality for Tephra. > Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra. > Task 4: Implement Omid required features for Phoenix: > Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction > where every write occurs after the checkpoint is not visible by the > transaction. Explanations for this feature can be seen in [TEPHRA-96]. > Task 4.2: Add an option to mark a key as non-conflicting. The motivation is > to reduce the size of the write set needed by the transaction manager upon > commit as well as reduce the conflict detection work. > Task 4.3: Add support for transactions that never abort. Such transactions > will only make other inflight transactions abort and will abort only in case > of a transaction manager failure. > These transactions are needed for ‘create index’ and the scenario was > discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind > of transactions was also discussed in [OMID-56]. > Task 4.4: Add support for returning multiple versions in a scan. The use case > is described in [TEPHRA-134]. > Task 4.5: Change Omid's timestamp mechanism to return real time based > timestamp, while keeping monotonicity. > Task 5: Implement TAL functionality for Omid. > Task 6: Implement performance tests and tune Omid for Phoenix use. This task > requires understanding of common usage scenarios in Phoenix as well as > defining the tradeoff between throughput and latency. > Could you please review the proposed work plan? > Also, could you please let me know whether I missed any augmentation needed > for Omid in order to support Phoenix operations? > I opened a jira [OMID-82] that encapsulates all Omid related development for > Phoenix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495958#comment-16495958
]
James Taylor commented on PHOENIX-3623:
---
Latest test run on my Linux box. Would be good to get your help with these,
[~ohads]:
* Not seeing the Invalid URI for NameNode address issue so hopefully that's
just a spurious issue.
* Still seeing an issue with deletes of rows in local index:
LocalImmutableTxIndexIT>BaseIndexIT.testDeleteFromAllPKColumnIndex:203
expected:<3> but was:<0>
* Also seeing issue with rollback of deletes (might be related to above):
TxCheckpointIT.testRollbackOfUncommittedDeleteSingleCol:123->testRollbackOfUncommittedDelete:185
expected: but was:
* Phoenix isn't dealing with DDL/DML fence correctly. Not sure how. See my
comment above.
* A few occurrences of this exception:
java.lang.IllegalArgumentException: Timestamp not allowed in transactional user
operations
* Other issues are test-only issues and can be dealt with when we parameterize
for both Tephra and Omid
{code:java}
[INFO] --- maven-failsafe-plugin:2.20:integration-test
(ParallelStatsEnabledTest) @ phoenix-core ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.phoenix.coprocessor.StatisticsCollectionRunTrackerIT
[INFO] Running org.apache.phoenix.end2end.MultiCfQueryExecIT
[INFO] Running org.apache.phoenix.end2end.KeyOnlyIT
[INFO] Running org.apache.phoenix.end2end.ExplainPlanWithStatsEnabledIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.995 s
- in org.apache.phoenix.coprocessor.StatisticsCollectionRunTrackerIT
[INFO] Running org.apache.phoenix.end2end.ParallelIteratorsIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.089 s
- in org.apache.phoenix.end2end.KeyOnlyIT
[INFO] Running org.apache.phoenix.end2end.QueryWithTableSampleIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.913 s
- in org.apache.phoenix.end2end.ParallelIteratorsIT
[INFO] Running org.apache.phoenix.end2end.ReadIsolationLevelIT
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 44.447
s - in org.apache.phoenix.end2end.MultiCfQueryExecIT
[INFO] Running org.apache.phoenix.end2end.SaltedViewIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.818 s
- in org.apache.phoenix.end2end.ReadIsolationLevelIT
[INFO] Running org.apache.phoenix.end2end.TenantSpecificTablesDDLIT
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 54.841
s <<< FAILURE! - in org.apache.phoenix.end2end.SaltedViewIT
[ERROR] testSaltedUpdatableViewWithLocalIndex[transactional =
true](org.apache.phoenix.end2end.SaltedViewIT) Time elapsed: 12.23 s <<<
FAILURE!
java.lang.AssertionError: expected:<8> but was:<4>
at
org.apache.phoenix.end2end.SaltedViewIT.testSaltedUpdatableViewWithLocalIndex(SaltedViewIT.java:43)
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 84.208
s - in org.apache.phoenix.end2end.QueryWithTableSampleIT
[INFO] Running org.apache.phoenix.end2end.TransactionalViewIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.822 s
- in org.apache.phoenix.end2end.TransactionalViewIT
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 109.381
s - in org.apache.phoenix.end2end.ExplainPlanWithStatsEnabledIT
[INFO] Running org.apache.phoenix.end2end.TenantSpecificTablesDMLIT
[INFO] Running org.apache.phoenix.end2end.ViewIT
[INFO] Running org.apache.phoenix.end2end.index.ImmutableIndexWithStatsIT
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.626 s
- in org.apache.phoenix.end2end.index.ImmutableIndexWithStatsIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 164.113
s - in org.apache.phoenix.end2end.TenantSpecificTablesDDLIT
[INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 170.16
s - in org.apache.phoenix.end2end.TenantSpecificTablesDMLIT
[ERROR] Tests run: 56, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 340.83
s <<< FAILURE! - in org.apache.phoenix.end2end.ViewIT
[ERROR] testNonSaltedUpdatableViewWithLocalIndex[transactional =
true](org.apache.phoenix.end2end.ViewIT) Time elapsed: 10.385 s <<< FAILURE!
java.lang.AssertionError: expected:<6> but was:<2>
at
org.apache.phoenix.end2end.ViewIT.testNonSaltedUpdatableViewWithLocalIndex(ViewIT.java:140)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]
SaltedViewIT.testSaltedUpdatableViewWithLocalIndex:43->BaseViewIT.testUpdatableViewWithIndex:81->BaseViewIT.testUpdatableViewIndex:180
expected:<8> but was:<4>
[ERROR]
ViewIT.testNonSaltedUpdatableViewWithLocalIndex:140->BaseViewIT.testUpdatableViewWithIndex:81->BaseViewIT.testUpdatableViewIndex:180
expected:<6> but was:<2>
[INFO]
[ERROR] Tests run:
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495420#comment-16495420 ] James Taylor commented on PHOENIX-3623: --- The failure in GlobalImmutableTxIndexIT.testCreateIndexAfterUpsertStarted() is due to the difference in how Tephra and Omid handle the DML/DDL fence. With Tephra, we get a conflict detected exception at commit time that we can react too. How should the client deal with this fence in Omid? The reason that the client needs to know is that it needs to re-submit the transaction so that the second time the rows for the secondary index will be added. The first time of the commit it doesn't know that there's an index (but the exception occurs so that it can retry). How should we handle this in Phoenix with Omid, [~ohads]? > Integrate Omid with Phoenix > --- > > Key: PHOENIX-3623 > URL: https://issues.apache.org/jira/browse/PHOENIX-3623 > Project: Phoenix > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Ohad Shacham >Priority: Major > > The purpose of this Jira is to propose a work plan for connecting Omid to > Phoenix. > Each task of the following will be handled in a seperate sub Jira. Subtasks > 4.* are related to augmenting Omid to support features required by Phoenix > and therefore, their corresponding Jiras will appear under Omid and not under > Phoenix. > Each task is completed by a commit. > Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls > are integrated inside Phoenix code. Therefore, in order to support both Omid > and Tephra, we need to add another abstraction layer that later-on will be > connected to both Tephra and Omid. The first tasks is to define such an > interface. > Task 2: Implement TAL functionality for Tephra. > Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra. > Task 4: Implement Omid required features for Phoenix: > Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction > where every write occurs after the checkpoint is not visible by the > transaction. Explanations for this feature can be seen in [TEPHRA-96]. > Task 4.2: Add an option to mark a key as non-conflicting. The motivation is > to reduce the size of the write set needed by the transaction manager upon > commit as well as reduce the conflict detection work. > Task 4.3: Add support for transactions that never abort. Such transactions > will only make other inflight transactions abort and will abort only in case > of a transaction manager failure. > These transactions are needed for ‘create index’ and the scenario was > discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind > of transactions was also discussed in [OMID-56]. > Task 4.4: Add support for returning multiple versions in a scan. The use case > is described in [TEPHRA-134]. > Task 4.5: Change Omid's timestamp mechanism to return real time based > timestamp, while keeping monotonicity. > Task 5: Implement TAL functionality for Omid. > Task 6: Implement performance tests and tune Omid for Phoenix use. This task > requires understanding of common usage scenarios in Phoenix as well as > defining the tradeoff between throughput and latency. > Could you please review the proposed work plan? > Also, could you please let me know whether I missed any augmentation needed > for Omid in order to support Phoenix operations? > I opened a jira [OMID-82] that encapsulates all Omid related development for > Phoenix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[
https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495380#comment-16495380
]
James Taylor commented on PHOENIX-3623:
---
With the change to have local indexes maintained from the client, the test run
looks similar to the above which is good. Here are some particular tests that
may indicate real issues:
* java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid URI
for NameNode address (check fs.defaultFS): file:/ has no authority. Any idea
where this is from and how to prevent it, [~ohads]?
* The failure in GlobalImmutableTxIndexIT.testCreateIndexAfterUpsertStarted()
looks real. This test creates an index from a separate connection while the
other connection upserts a new row. The original connection commit should
detect that the index was added and rerun the transaction so that the new row
is added to the index (but this isn't happening for Omid). Would you have time
to take a look, [~ohads]?
* The failure here looks like there may be an issue with deletes:
testDeleteFromAllPKColumnIndex[LocalImmutableTxIndexIT_localIndex=true,mutable=false,transactional=true,columnEncoded=false]
* The failures that complain about the byte count being different are easily
fixed (these are due to Omid's shadow cells).
ColumnEncodedMutableTxStatsCollectorIT>StatsCollectorIT.testWithMultiCF:533
expected:<... 26-CHUNK 25 ROWS 12[53]0 BYTES PARALLEL 1-W...> but was:<...
26-CHUNK 25 ROWS 12[42]0 BYTES PARALLEL 1-W...>
I'll commit what I have since it gets us closer.
{code:java}
[INFO] --- maven-failsafe-plugin:2.20:integration-test
(ParallelStatsEnabledTest) @ phoenix-core ---
[INFO]
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.phoenix.end2end.KeyOnlyIT
[INFO] Running org.apache.phoenix.end2end.MultiCfQueryExecIT
[INFO] Running org.apache.phoenix.coprocessor.StatisticsCollectionRunTrackerIT
[INFO] Running org.apache.phoenix.end2end.ExplainPlanWithStatsEnabledIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.713 s
- in org.apache.phoenix.end2end.KeyOnlyIT
[INFO] Running org.apache.phoenix.end2end.ParallelIteratorsIT
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 20.244 s
- in org.apache.phoenix.coprocessor.StatisticsCollectionRunTrackerIT
[INFO] Running org.apache.phoenix.end2end.QueryWithTableSampleIT
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.683 s
- in org.apache.phoenix.end2end.ParallelIteratorsIT
[INFO] Running org.apache.phoenix.end2end.ReadIsolationLevelIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.653 s
- in org.apache.phoenix.end2end.ReadIsolationLevelIT
[INFO] Running org.apache.phoenix.end2end.SaltedViewIT
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 32.499
s - in org.apache.phoenix.end2end.MultiCfQueryExecIT
[INFO] Running org.apache.phoenix.end2end.TenantSpecificTablesDDLIT
[ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 41.297
s <<< FAILURE! - in org.apache.phoenix.end2end.SaltedViewIT
[ERROR] testSaltedUpdatableViewWithLocalIndex[transactional =
true](org.apache.phoenix.end2end.SaltedViewIT) Time elapsed: 12.114 s <<<
FAILURE!
java.lang.AssertionError: expected:<8> but was:<4>
at
org.apache.phoenix.end2end.SaltedViewIT.testSaltedUpdatableViewWithLocalIndex(SaltedViewIT.java:43)
[INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 56.17 s
- in org.apache.phoenix.end2end.QueryWithTableSampleIT
[INFO] Running org.apache.phoenix.end2end.TransactionalViewIT
[INFO] Tests run: 28, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.525
s - in org.apache.phoenix.end2end.ExplainPlanWithStatsEnabledIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.447 s
- in org.apache.phoenix.end2end.TransactionalViewIT
[INFO] Running org.apache.phoenix.end2end.TenantSpecificTablesDMLIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0 s <<<
FAILURE! - in org.apache.phoenix.end2end.TenantSpecificTablesDMLIT
[ERROR] org.apache.phoenix.end2end.TenantSpecificTablesDMLIT Time elapsed: 0 s
<<< ERROR!
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid URI for
NameNode address (check fs.defaultFS): file:/ has no authority.
Caused by: java.lang.IllegalArgumentException: Invalid URI for NameNode address
(check fs.defaultFS): file:/ has no authority.
[INFO] Running org.apache.phoenix.end2end.ViewIT
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.002 s
<<< FAILURE! - in org.apache.phoenix.end2end.ViewIT
[ERROR] org.apache.phoenix.end2end.ViewIT Time elapsed: 0.002 s <<< ERROR!
java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid URI for
NameNode address
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490880#comment-16490880 ] James Taylor commented on PHOENIX-3623: --- Our [storage format|https://phoenix.apache.org/columnencoding.html] that is used by default in Phoenix won't work well out-of-the-box with Omid. In the mutable case, it expects that the column qualifier can be used as the position in the List used by HBase, and in the immutable case, it expects a single Cell to be used in which all values are stored. We can try to support these storage formats in the future, but for now I can disable them when we use Omid transactions. That should help reduce the failures substantially. > Integrate Omid with Phoenix > --- > > Key: PHOENIX-3623 > URL: https://issues.apache.org/jira/browse/PHOENIX-3623 > Project: Phoenix > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Ohad Shacham >Priority: Major > > The purpose of this Jira is to propose a work plan for connecting Omid to > Phoenix. > Each task of the following will be handled in a seperate sub Jira. Subtasks > 4.* are related to augmenting Omid to support features required by Phoenix > and therefore, their corresponding Jiras will appear under Omid and not under > Phoenix. > Each task is completed by a commit. > Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls > are integrated inside Phoenix code. Therefore, in order to support both Omid > and Tephra, we need to add another abstraction layer that later-on will be > connected to both Tephra and Omid. The first tasks is to define such an > interface. > Task 2: Implement TAL functionality for Tephra. > Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra. > Task 4: Implement Omid required features for Phoenix: > Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction > where every write occurs after the checkpoint is not visible by the > transaction. Explanations for this feature can be seen in [TEPHRA-96]. > Task 4.2: Add an option to mark a key as non-conflicting. The motivation is > to reduce the size of the write set needed by the transaction manager upon > commit as well as reduce the conflict detection work. > Task 4.3: Add support for transactions that never abort. Such transactions > will only make other inflight transactions abort and will abort only in case > of a transaction manager failure. > These transactions are needed for ‘create index’ and the scenario was > discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind > of transactions was also discussed in [OMID-56]. > Task 4.4: Add support for returning multiple versions in a scan. The use case > is described in [TEPHRA-134]. > Task 4.5: Change Omid's timestamp mechanism to return real time based > timestamp, while keeping monotonicity. > Task 5: Implement TAL functionality for Omid. > Task 6: Implement performance tests and tune Omid for Phoenix use. This task > requires understanding of common usage scenarios in Phoenix as well as > defining the tradeoff between throughput and latency. > Could you please review the proposed work plan? > Also, could you please let me know whether I missed any augmentation needed > for Omid in order to support Phoenix operations? > I opened a jira [OMID-82] that encapsulates all Omid related development for > Phoenix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924643#comment-15924643 ] Thomas D'Silva commented on PHOENIX-3623: - We only checkpoint if we are querying the same table that we are writing to and we have uncommitted data (for eg. UPSERT SELECT into the same table). Once we checkpoint, following operations do not always have to checkpoint. Calling setVisibility does not checkpoint. > Integrate Omid with Phoenix > --- > > Key: PHOENIX-3623 > URL: https://issues.apache.org/jira/browse/PHOENIX-3623 > Project: Phoenix > Issue Type: New Feature >Reporter: Ohad Shacham >Assignee: Ohad Shacham > > The purpose of this Jira is to propose a work plan for connecting Omid to > Phoenix. > Each task of the following will be handled in a seperate sub Jira. Subtasks > 4.* are related to augmenting Omid to support features required by Phoenix > and therefore, their corresponding Jiras will appear under Omid and not under > Phoenix. > Each task is completed by a commit. > Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls > are integrated inside Phoenix code. Therefore, in order to support both Omid > and Tephra, we need to add another abstraction layer that later-on will be > connected to both Tephra and Omid. The first tasks is to define such an > interface. > Task 2: Implement TAL functionality for Tephra. > Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra. > Task 4: Implement Omid required features for Phoenix: > Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction > where every write occurs after the checkpoint is not visible by the > transaction. Explanations for this feature can be seen in [TEPHRA-96]. > Task 4.2: Add an option to mark a key as non-conflicting. The motivation is > to reduce the size of the write set needed by the transaction manager upon > commit as well as reduce the conflict detection work. > Task 4.3: Add support for transactions that never abort. Such transactions > will only make other inflight transactions abort and will abort only in case > of a transaction manager failure. > These transactions are needed for ‘create index’ and the scenario was > discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind > of transactions was also discussed in [OMID-56]. > Task 4.4: Add support for returning multiple versions in a scan. The use case > is described in [TEPHRA-134]. > Task 4.5: Change Omid's timestamp mechanism to return real time based > timestamp, while keeping monotonicity. > Task 5: Implement TAL functionality for Omid. > Task 6: Implement performance tests and tune Omid for Phoenix use. This task > requires understanding of common usage scenarios in Phoenix as well as > defining the tradeoff between throughput and latency. > Could you please review the proposed work plan? > Also, could you please let me know whether I missed any augmentation needed > for Omid in order to support Phoenix operations? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862743#comment-15862743 ] Ohad Shacham commented on PHOENIX-3623: --- HI [~giacomotaylor] Sub-jira [PHOENIX-3656], for Task 1, was added and a code review was uploaded to https://reviews.apache.org/r/56498/ Thx, Ohad > Integrate Omid with Phoenix > --- > > Key: PHOENIX-3623 > URL: https://issues.apache.org/jira/browse/PHOENIX-3623 > Project: Phoenix > Issue Type: New Feature >Reporter: Ohad Shacham > > The purpose of this Jira is to propose a work plan for connecting Omid to > Phoenix. > Each task of the following will be handled in a seperate sub Jira. Subtasks > 4.* are related to augmenting Omid to support features required by Phoenix > and therefore, their corresponding Jiras will appear under Omid and not under > Phoenix. > Each task is completed by a commit. > Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls > are integrated inside Phoenix code. Therefore, in order to support both Omid > and Tephra, we need to add another abstraction layer that later-on will be > connected to both Tephra and Omid. The first tasks is to define such an > interface. > Task 2: Implement TAL functionality for Tephra. > Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra. > Task 4: Implement Omid required features for Phoenix: > Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction > where every write occurs after the checkpoint is not visible by the > transaction. Explanations for this feature can be seen in [TEPHRA-96]. > Task 4.2: Add an option to mark a key as non-conflicting. The motivation is > to reduce the size of the write set needed by the transaction manager upon > commit as well as reduce the conflict detection work. > Task 4.3: Add support for transactions that never abort. Such transactions > will only make other inflight transactions abort and will abort only in case > of a transaction manager failure. > These transactions are needed for ‘create index’ and the scenario was > discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind > of transactions was also discussed in [OMID-56]. > Task 4.4: Add support for returning multiple versions in a scan. The use case > is described in [TEPHRA-134]. > Task 5: Implement TAL functionality for Omid. > Task 6: Implement performance tests and tune Omid for Phoenix use. This task > requires understanding of common usage scenarios in Phoenix as well as > defining the tradeoff between throughput and latency. > Could you please review the proposed work plan? > Also, could you please let me know whether I missed any augmentation needed > for Omid in order to support Phoenix operations? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (PHOENIX-3623) Integrate Omid with Phoenix
[ https://issues.apache.org/jira/browse/PHOENIX-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849672#comment-15849672 ] Ohad Shacham commented on PHOENIX-3623: --- Hi [~giacomotaylor], I added a task at the end for implementing performance tests and tune Omid for Phoenix common usages. Currently, Omid design is throughput oriented, however, can be tuned to reduce the latency. We should define the tradeoff between these two and tune Omid according to the common load that the TPS will have. > Integrate Omid with Phoenix > --- > > Key: PHOENIX-3623 > URL: https://issues.apache.org/jira/browse/PHOENIX-3623 > Project: Phoenix > Issue Type: New Feature >Reporter: Ohad Shacham > > The purpose of this Jira is to propose a work plan for connecting Omid to > Phoenix. > Each task of the following will be handled in a seperate sub Jira. Subtasks > 4.* are related to augmenting Omid to support features required by Phoenix > and therefore, their corresponding Jiras will appear under Omid and not under > Phoenix. > Each task is completed by a commit. > Task 1: Adding transaction abstraction layer (TAL) - Currently Tephra calls > are integrated inside Phoenix code. Therefore, in order to support both Omid > and Tephra, we need to add another abstraction layer that later-on will be > connected to both Tephra and Omid. The first tasks is to define such an > interface. > Task 2: Implement TAL functionality for Tephra. > Task 3: Refactor Phoenix to use TAL instead of direct calls to Tephra. > Task 4: Implement Omid required features for Phoenix: > Task 4.1: Add checkpoints to Omid. A checkpoint is a point in a transaction > where every write occurs after the checkpoint is not visible by the > transaction. Explanations for this feature can be seen in [TEPHRA-96]. > Task 4.2: Add an option to mark a key as non-conflicting. The motivation is > to reduce the size of the write set needed by the transaction manager upon > commit as well as reduce the conflict detection work. > Task 4.3: Add support for transactions that never abort. Such transactions > will only make other inflight transactions abort and will abort only in case > of a transaction manager failure. > These transactions are needed for ‘create index’ and the scenario was > discussed in [TEPHRA-157] and [PHOENIX-2478]. Augmenting Omid with this kind > of transactions was also discussed in [OMID-56]. > Task 4.4: Add support for returning multiple versions in a scan. The use case > is described in [TEPHRA-134]. > Task 5: Implement TAL functionality for Omid. > Task 6: Implement performance tests and tune Omid for Phoenix use. This task > requires understanding of common usage scenarios in Phoenix as well as > defining the tradeoff between throughput and latency. > Could you please review the proposed work plan? > Also, could you please let me know whether I missed any augmentation needed > for Omid in order to support Phoenix operations? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
