[jira] [Commented] (HIVE-15999) Fix flakiness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883328#comment-15883328
]
Eugene Koifman commented on HIVE-15999:
---
+1
> Fix flakiness in TestDbTxnManager2
> --
>
> Key: HIVE-15999
> URL: https://issues.apache.org/jira/browse/HIVE-15999
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15999.1.patch
>
>
> Right now there is test flakiness wrt. TestDbTxnManager2. The error is like
> this:
> {code}
> java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:75)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.setUp(TestDbTxnManager2.java:90)
> {code}
> The failure is due to HiveConf used in the test being polluted by some test,
> e.g. in testDummyTxnManagerOnAcidTable(), conf entry HIVE_TXN_MANAGER is set
> to "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager" but not switched back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15999) Fix flakiness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877617#comment-15877617
]
Wei Zheng commented on HIVE-15999:
--
I cannot find a good answer. As we discussed, this might be a derby bug.
One thing I noticed that is different for TestDbTxnManager2 from other tests is
that, all other tests call "TxnDbUtil.setConfValues(conf);" before calling
"TxnDbUtil.prepDb();", but TestDbTxnManager2 only calls
"TxnDbUtil.setConfValues(conf);" once in a @BeforeClass method. By changing the
@BeforeClass method into constructor, it's guaranteed to be run for every UT,
which is consistent to all other tests.
I ran the ptest several times, and didn't see such failure anymore with the fix.
> Fix flakiness in TestDbTxnManager2
> --
>
> Key: HIVE-15999
> URL: https://issues.apache.org/jira/browse/HIVE-15999
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15999.1.patch
>
>
> Right now there is test flakiness wrt. TestDbTxnManager2. The error is like
> this:
> {code}
> java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:75)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.setUp(TestDbTxnManager2.java:90)
> {code}
> The failure is due to HiveConf used in the test being polluted by some test,
> e.g. in testDummyTxnManagerOnAcidTable(), conf entry HIVE_TXN_MANAGER is set
> to "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager" but not switched back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15999) Fix flakiness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877475#comment-15877475
]
Hive QA commented on HIVE-15999:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853815/HIVE-15999.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10252 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23]
(batchId=223)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=211)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3682/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3682/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3682/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853815 - PreCommit-HIVE-Build
> Fix flakiness in TestDbTxnManager2
> --
>
> Key: HIVE-15999
> URL: https://issues.apache.org/jira/browse/HIVE-15999
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15999.1.patch
>
>
> Right now there is test flakiness wrt. TestDbTxnManager2. The error is like
> this:
> {code}
> java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:75)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.setUp(TestDbTxnManager2.java:90)
> {code}
> The failure is due to HiveConf used in the test being polluted by some test,
> e.g. in testDummyTxnManagerOnAcidTable(), conf entry HIVE_TXN_MANAGER is set
> to "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager" but not switched back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15999) Fix flakiness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877258#comment-15877258
]
Eugene Koifman commented on HIVE-15999:
---
could you explain why this is causing a derby error?
> Fix flakiness in TestDbTxnManager2
> --
>
> Key: HIVE-15999
> URL: https://issues.apache.org/jira/browse/HIVE-15999
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15999.1.patch
>
>
> Right now there is test flakiness wrt. TestDbTxnManager2. The error is like
> this:
> {code}
> java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:75)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.setUp(TestDbTxnManager2.java:90)
> {code}
> The failure is due to HiveConf used in the test being polluted by some test,
> e.g. in testDummyTxnManagerOnAcidTable(), conf entry HIVE_TXN_MANAGER is set
> to "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager" but not switched back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15999) Fix flakiness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877225#comment-15877225
]
Wei Zheng commented on HIVE-15999:
--
[~ekoifman] Can you take a look please?
> Fix flakiness in TestDbTxnManager2
> --
>
> Key: HIVE-15999
> URL: https://issues.apache.org/jira/browse/HIVE-15999
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15999.1.patch
>
>
> Right now there is test flakiness wrt. TestDbTxnManager2. The error is like
> this:
> {code}
> java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:75)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.setUp(TestDbTxnManager2.java:90)
> {code}
> The failure is due to HiveConf used in the test being polluted by some test,
> e.g. in testDummyTxnManagerOnAcidTable(), conf entry HIVE_TXN_MANAGER is set
> to "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager" but not switched back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15999) Fix flakiness in TestDbTxnManager2
[
https://issues.apache.org/jira/browse/HIVE-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877220#comment-15877220
]
Hive QA commented on HIVE-15999:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12853815/HIVE-15999.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10251 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys]
(batchId=159)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag[0]
(batchId=173)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteDecimalXY
(batchId=173)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteTimestamp
(batchId=173)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3678/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3678/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3678/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12853815 - PreCommit-HIVE-Build
> Fix flakiness in TestDbTxnManager2
> --
>
> Key: HIVE-15999
> URL: https://issues.apache.org/jira/browse/HIVE-15999
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.2.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-15999.1.patch
>
>
> Right now there is test flakiness wrt. TestDbTxnManager2. The error is like
> this:
> {code}
> java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
> at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:75)
> at
> org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.setUp(TestDbTxnManager2.java:90)
> {code}
> The failure is due to HiveConf used in the test being polluted by some test,
> e.g. in testDummyTxnManagerOnAcidTable(), conf entry HIVE_TXN_MANAGER is set
> to "org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager" but not switched back.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
