[
https://issues.apache.org/jira/browse/HIVE-30250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-30250:
---------------------------------
Description:
Compactor tests that extend {{CompactorTest}} can fail depending on which other
test classes ran before them in the same Maven run. For example,
{{TestWorker2#minorNoBaseLotsOfDeltas}} fails with:
{code:java}
java.lang.AssertionError: expected:<12> but was:<15>
at
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.compactNoBaseLotsOfDeltas(TestWorker.java:644)
{code}
Cause
{{CompactorTest.setup()}} creates each test's directory as
{{{}${test.tmp.dir}/compactor_test_table_<N>{}}}, where {{N}} comes from a
static {{{}AtomicInteger{}}}. Surefire runs each test class in a new JVM
({{{}reuseForks=false{}}}), so {{N}} starts at 0 in every class.
{{ql/target/tmp}} is only cleaned once per Maven run, so tests in different
classes get the same directory.
{{TestCompactionMetrics}} never calls {{{}compactorTestCleanup(){}}}, so its
files stay behind. When a later test reuses that directory and the same table
name ({{{}mapwb{}}}), it finds old base and delta directories. The compactor
then marks some deltas as obsolete, and the test's file count assertions fail.
Reproduce
{code:java}
mvn -pl ql test
-Dtest='TestCompactionMetrics#testWorkerPerfMetrics,TestWorker2#minorNoBaseLotsOfDeltas'
-Dsurefire.runOrder=alphabetical
{code}
I recommend the following fix:
- {{{}CompactorTest.setup(){}}}: delete the directory before creating it, so
every test starts empty.
- {{{}TestCompactionMetrics{}}}: add an {{@After}} that calls
{{{}compactorTestCleanup(){}}}.
> Compactor tests are flaky because test classes reuse each other's temp
> directories
> ----------------------------------------------------------------------------------
>
> Key: HIVE-30250
> URL: https://issues.apache.org/jira/browse/HIVE-30250
> Project: Hive
> Issue Type: Test
> Components: Test
> Reporter: Marta Kuczora
> Assignee: Marta Kuczora
> Priority: Minor
>
> Compactor tests that extend {{CompactorTest}} can fail depending on which
> other test classes ran before them in the same Maven run. For example,
> {{TestWorker2#minorNoBaseLotsOfDeltas}} fails with:
> {code:java}
> java.lang.AssertionError: expected:<12> but was:<15>
> at
> org.apache.hadoop.hive.ql.txn.compactor.TestWorker.compactNoBaseLotsOfDeltas(TestWorker.java:644)
> {code}
> Cause
> {{CompactorTest.setup()}} creates each test's directory as
> {{{}${test.tmp.dir}/compactor_test_table_<N>{}}}, where {{N}} comes from a
> static {{{}AtomicInteger{}}}. Surefire runs each test class in a new JVM
> ({{{}reuseForks=false{}}}), so {{N}} starts at 0 in every class.
> {{ql/target/tmp}} is only cleaned once per Maven run, so tests in different
> classes get the same directory.
> {{TestCompactionMetrics}} never calls {{{}compactorTestCleanup(){}}}, so its
> files stay behind. When a later test reuses that directory and the same table
> name ({{{}mapwb{}}}), it finds old base and delta directories. The compactor
> then marks some deltas as obsolete, and the test's file count assertions fail.
> Reproduce
> {code:java}
> mvn -pl ql test
> -Dtest='TestCompactionMetrics#testWorkerPerfMetrics,TestWorker2#minorNoBaseLotsOfDeltas'
> -Dsurefire.runOrder=alphabetical
> {code}
> I recommend the following fix:
> - {{{}CompactorTest.setup(){}}}: delete the directory before creating it, so
> every test starts empty.
> - {{{}TestCompactionMetrics{}}}: add an {{@After}} that calls
> {{{}compactorTestCleanup(){}}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)