So the issue here is, somehow Test*ExportSnapshot (4 tests classes) tests
are stepping on each other.
When we run locally, tests are run sequentially, hence they always pass.
But in precommit/flaky-tests-job, there are multiple forks and multiple
tests can run in parallel. That's when they fail.
If you look at this failure [1], the following exception is weird because
these tests should be using /tmp/* dir, but one of our
<module>/target/test-data/xxxx-yyyy random dirs.
2017-04-21 21:15:48,787 WARN [IPC Server handler 3 on 46240]
localizer.ResourceLocalizationService$LocalizerRunner(1015): {
file:/tmp/hadoop-yarn/staging/jenkins/.staging/job_1492809330233_0001/job.split,
1492809346000, FILE, null } failed: File
file:/tmp/hadoop-yarn/staging/jenkins/.staging/job_1492809330233_0001/job.split
does not exist
java.io.FileNotFoundException: File
file:/tmp/hadoop-yarn/staging/jenkins/.staging/job_1492809330233_0001/job.split
does not exist
at
org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[1]
https://builds.apache.org/job/HBASE-Flaky-Tests/15333/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot-output.txt/*view*/
On Wed, Apr 26, 2017 at 2:39 PM, Stephen Jiang <[email protected]>
wrote:
> It is not flaky in my local machine :-(. Today I can 100% repro in 7 runs.
> "org.apache.hadoop.hbase.snapshot.TestExportSnapshot" also always fails.
>
> Thanks
> Stephen
>
> On Wed, Apr 26, 2017 at 2:32 PM, Ted Yu <[email protected]> wrote:
>
> > You can take a look at:
> > https://builds.apache.org/job/HBASE-Find-Flaky-Tests/
> > lastSuccessfulBuild/artifact/dashboard.html
> >
> > It seems the test is flaky.
> >
> > FYI
> >
> > On Wed, Apr 26, 2017 at 2:25 PM, Stephen Jiang <[email protected]>
> > wrote:
> >
> > > I am running "org.apache.hadoop.hbase.snapshot.
> TestSecureExportSnapshot"
> > > test in my local branch-1 today and it failed. I just wonder anyone
> > else
> > > see the same failure?
> > >
> > > -------------------------------------------------------
> > > T E S T S
> > > -------------------------------------------------------
> > > Running org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot
> > > Tests run: 9, Failures: 7, Errors: 0, Skipped: 0, Time elapsed: 163.004
> > sec
> > > <<< FAILURE! - in org.apache.hadoop.hbase.snapshot.
> > > TestSecureExportSnapshot
> > > testExportWithTargetName(org.apache.hadoop.hbase.snapshot.
> > > TestSecureExportSnapshot)
> > > Time elapsed: 14.708 sec <<< FAILURE!
> > > java.lang.AssertionError: expected:<0> but was:<1>
> > > at org.junit.Assert.fail(Assert.java:88)
> > > at org.junit.Assert.failNotEquals(Assert.java:834)
> > > at org.junit.Assert.assertEquals(Assert.java:645)
> > > at org.junit.Assert.assertEquals(Assert.java:631)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:294)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:266)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportWithTargetName(TestExportSnapshot.java:223)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > > FrameworkMethod.java:50)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(
> > > ReflectiveCallable.java:12)
> > > at
> > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > > FrameworkMethod.java:47)
> > > at
> > > org.junit.internal.runners.statements.InvokeMethod.
> > > evaluate(InvokeMethod.java:17)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.
> > > evaluate(RunBefores.java:26)
> > > at
> > > org.junit.internal.runners.statements.RunAfters.evaluate(
> > > RunAfters.java:27)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:298)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:292)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > testExportFileSystemState(org.apache.hadoop.hbase.snapshot.
> > > TestSecureExportSnapshot)
> > > Time elapsed: 13.045 sec <<< FAILURE!
> > > java.lang.AssertionError: expected:<0> but was:<1>
> > > at org.junit.Assert.fail(Assert.java:88)
> > > at org.junit.Assert.failNotEquals(Assert.java:834)
> > > at org.junit.Assert.assertEquals(Assert.java:645)
> > > at org.junit.Assert.assertEquals(Assert.java:631)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:294)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:266)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:198)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > > FrameworkMethod.java:50)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(
> > > ReflectiveCallable.java:12)
> > > at
> > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > > FrameworkMethod.java:47)
> > > at
> > > org.junit.internal.runners.statements.InvokeMethod.
> > > evaluate(InvokeMethod.java:17)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.
> > > evaluate(RunBefores.java:26)
> > > at
> > > org.junit.internal.runners.statements.RunAfters.evaluate(
> > > RunAfters.java:27)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:298)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:292)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > testConsecutiveExports(org.apache.hadoop.hbase.snapshot.
> > > TestSecureExportSnapshot)
> > > Time elapsed: 12.967 sec <<< FAILURE!
> > > java.lang.AssertionError: expected:<0> but was:<1>
> > > at org.junit.Assert.fail(Assert.java:88)
> > > at org.junit.Assert.failNotEquals(Assert.java:834)
> > > at org.junit.Assert.assertEquals(Assert.java:645)
> > > at org.junit.Assert.assertEquals(Assert.java:631)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:294)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testConsecutiveExports(TestExportSnapshot.java:215)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > > FrameworkMethod.java:50)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(
> > > ReflectiveCallable.java:12)
> > > at
> > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > > FrameworkMethod.java:47)
> > > at
> > > org.junit.internal.runners.statements.InvokeMethod.
> > > evaluate(InvokeMethod.java:17)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.
> > > evaluate(RunBefores.java:26)
> > > at
> > > org.junit.internal.runners.statements.RunAfters.evaluate(
> > > RunAfters.java:27)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:298)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:292)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > testEmptyExportFileSystemState(org.apache.hadoop.hbase.snapshot.
> > > TestSecureExportSnapshot)
> > > Time elapsed: 12.913 sec <<< FAILURE!
> > > java.lang.AssertionError: expected:<0> but was:<1>
> > > at org.junit.Assert.fail(Assert.java:88)
> > > at org.junit.Assert.failNotEquals(Assert.java:834)
> > > at org.junit.Assert.assertEquals(Assert.java:645)
> > > at org.junit.Assert.assertEquals(Assert.java:631)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:294)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:266)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testEmptyExportFileSystemState(TestExportSnapshot.java:209)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > > FrameworkMethod.java:50)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(
> > > ReflectiveCallable.java:12)
> > > at
> > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > > FrameworkMethod.java:47)
> > > at
> > > org.junit.internal.runners.statements.InvokeMethod.
> > > evaluate(InvokeMethod.java:17)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.
> > > evaluate(RunBefores.java:26)
> > > at
> > > org.junit.internal.runners.statements.RunAfters.evaluate(
> > > RunAfters.java:27)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:298)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:292)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > testSnapshotWithRefsExportFileSystemState(org.apache.hadoop.
> > > hbase.snapshot.TestSecureExportSnapshot)
> > > Time elapsed: 13.342 sec <<< FAILURE!
> > > java.lang.AssertionError: expected:<0> but was:<1>
> > > at org.junit.Assert.fail(Assert.java:88)
> > > at org.junit.Assert.failNotEquals(Assert.java:834)
> > > at org.junit.Assert.assertEquals(Assert.java:645)
> > > at org.junit.Assert.assertEquals(Assert.java:631)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:294)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:266)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:260)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:240)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > > FrameworkMethod.java:50)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(
> > > ReflectiveCallable.java:12)
> > > at
> > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > > FrameworkMethod.java:47)
> > > at
> > > org.junit.internal.runners.statements.InvokeMethod.
> > > evaluate(InvokeMethod.java:17)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.
> > > evaluate(RunBefores.java:26)
> > > at
> > > org.junit.internal.runners.statements.RunAfters.evaluate(
> > > RunAfters.java:27)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:298)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:292)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > testExportFileSystemStateWithSkipTmp(org.apache.hadoop.hbase.snapshot.
> > > TestSecureExportSnapshot)
> > > Time elapsed: 12.136 sec <<< FAILURE!
> > > java.lang.AssertionError: expected:<0> but was:<1>
> > > at org.junit.Assert.fail(Assert.java:88)
> > > at org.junit.Assert.failNotEquals(Assert.java:834)
> > > at org.junit.Assert.assertEquals(Assert.java:645)
> > > at org.junit.Assert.assertEquals(Assert.java:631)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:294)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemState(TestExportSnapshot.java:266)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.
> > > testExportFileSystemStateWithSkipTmp(TestExportSnapshot.java:204)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > > FrameworkMethod.java:50)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(
> > > ReflectiveCallable.java:12)
> > > at
> > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > > FrameworkMethod.java:47)
> > > at
> > > org.junit.internal.runners.statements.InvokeMethod.
> > > evaluate(InvokeMethod.java:17)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.
> > > evaluate(RunBefores.java:26)
> > > at
> > > org.junit.internal.runners.statements.RunAfters.evaluate(
> > > RunAfters.java:27)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:298)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:292)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > > testExportRetry(org.apache.hadoop.hbase.snapshot.
> > TestSecureExportSnapshot)
> > > Time elapsed: 12.903 sec <<< FAILURE!
> > > java.lang.AssertionError: expected:<0> but was:<1>
> > > at org.junit.Assert.fail(Assert.java:88)
> > > at org.junit.Assert.failNotEquals(Assert.java:834)
> > > at org.junit.Assert.assertEquals(Assert.java:645)
> > > at org.junit.Assert.assertEquals(Assert.java:631)
> > > at
> > > org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportRetry(
> > > TestExportSnapshot.java:329)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:
> > > 62)
> > > at
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
> > > FrameworkMethod.java:50)
> > > at
> > > org.junit.internal.runners.model.ReflectiveCallable.run(
> > > ReflectiveCallable.java:12)
> > > at
> > > org.junit.runners.model.FrameworkMethod.invokeExplosively(
> > > FrameworkMethod.java:47)
> > > at
> > > org.junit.internal.runners.statements.InvokeMethod.
> > > evaluate(InvokeMethod.java:17)
> > > at
> > > org.junit.internal.runners.statements.RunBefores.
> > > evaluate(RunBefores.java:26)
> > > at
> > > org.junit.internal.runners.statements.RunAfters.evaluate(
> > > RunAfters.java:27)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:298)
> > > at
> > > org.junit.internal.runners.statements.FailOnTimeout$
> > > CallableStatement.call(FailOnTimeout.java:292)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> > >
> > > Results :
> > >
> > > Failed tests:
> > >
> > > TestSecureExportSnapshot>TestExportSnapshot.
> testConsecutiveExports:215->
> > > TestExportSnapshot.testExportFileSystemState:294
> > > expected:<0> but was:<1>
> > >
> > > TestSecureExportSnapshot>TestExportSnapshot.
> > testEmptyExportFileSystemState
> > > :209->TestExportSnapshot.testExportFileSystemState:266-
> > > >TestExportSnapshot.testExportFileSystemState:294
> > > expected:<0> but was:<1>
> > >
> > > TestSecureExportSnapshot>TestExportSnapshot.
> > testExportFileSystemState:198-
> > > >TestExportSnapshot.testExportFileSystemState:266->TestExportSnapshot.
> > > testExportFileSystemState:294
> > > expected:<0> but was:<1>
> > >
> > > TestSecureExportSnapshot>TestExportSnapshot.
> > testExportFileSystemStateWithS
> > > kipTmp:204->TestExportSnapshot.testExportFileSystemState:266-
> > > >TestExportSnapshot.testExportFileSystemState:294
> > > expected:<0> but was:<1>
> > > TestSecureExportSnapshot>TestExportSnapshot.testExportRetry:329
> > > expected:<0> but was:<1>
> > >
> > > TestSecureExportSnapshot>TestExportSnapshot.
> > testExportWithTargetName:223->
> > > TestExportSnapshot.testExportFileSystemState:266->TestExportSnapshot.
> > > testExportFileSystemState:294
> > > expected:<0> but was:<1>
> > >
> > > TestSecureExportSnapshot>TestExportSnapshot.
> > testSnapshotWithRefsExportFile
> > > SystemState:240->TestExportSnapshot.testSnapshotWithRefsExportFile
> > > SystemState:260->TestExportSnapshot.testExportFileSystemState:266-
> > > >TestExportSnapshot.testExportFileSystemState:294
> > > expected:<0> but was:<1>
> > >
> > > Tests run: 9, Failures: 7, Errors: 0, Skipped: 0
> > >
> >
>
--
-- Appy
