[
https://issues.apache.org/jira/browse/OOZIE-2382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated OOZIE-2382:
---------------------------------
Attachment: OOZIE-2382.001.patch
The patch fixes the problem by resetting the {{PigStats}} as described above.
To verify this, I hacked {{testPig_withNullExternalId}} to first run one of the
other Pig tests to force the ordering. Without the fix, it would fail; with
the fix, it passes.
> rg.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID is flakey
> ------------------------------------------------------------------------------
>
> Key: OOZIE-2382
> URL: https://issues.apache.org/jira/browse/OOZIE-2382
> Project: Oozie
> Issue Type: Bug
> Components: tests
> Affects Versions: trunk
> Reporter: Robert Kanter
> Assignee: Robert Kanter
> Fix For: trunk
>
> Attachments: OOZIE-2382.001.patch
>
>
> {{rg.apache.oozie.action.hadoop.TestPigMain.testPig_withNullExternalID}} is
> flakey.
> {noformat}
> junit.framework.AssertionFailedError
> at junit.framework.Assert.fail(Assert.java:48)
> at junit.framework.Assert.assertTrue(Assert.java:20)
> at junit.framework.Assert.assertFalse(Assert.java:34)
> at junit.framework.Assert.assertFalse(Assert.java:41)
> at
> org.apache.oozie.action.hadoop.PigTestCase.testPig_withNullExternalID(PigTestCase.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:168)
> at junit.framework.TestCase.runBare(TestCase.java:134)
> at junit.framework.TestResult$1.protect(TestResult.java:110)
> at junit.framework.TestResult.runProtected(TestResult.java:128)
> at junit.framework.TestResult.run(TestResult.java:113)
> at junit.framework.TestCase.run(TestCase.java:124)
> at junit.framework.TestSuite.runTest(TestSuite.java:243)
> at junit.framework.TestSuite.run(TestSuite.java:238)
> at
> org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:24)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
> We've seen this for months but has been really tricky to track down. The
> output of the test has this:
> {noformat}
> Run pig script using PigRunner.run() for Pig version 0.8+
> Apache Pig version 0.12.0-cdh5.5.0-SNAPSHOT (rexported)
> compiled Sep 30 2015, 22:11:56
> Hadoop Job IDs executed by Pig: job_20151001151153510_0001
> Run pig script using PigRunner.run() for Pig version 0.8+
> {noformat}
> Compared with a successful run of the test, the difference is that it shows a
> Hadoop Job ID. This test is supposed to make Pig fail and not run any jobs,
> which can be confirmed in the test output, yet there's a Hadoop Job ID here!
> It turns out that {{PigRunner}} eventually gets to {{PigStats}}, which is a
> singleton. And because we're running all of the Pig tests in the same JVM,
> {{PigStats}} is actually leaking through the tests. It doesn't seem to be a
> problem most of the time because it must be getting overwritten at each run,
> however, because {{testPig_withNullExternalID}} is purposefully failing the
> Pig job, it's not, and we end up with the {{PigStats}} from the previous test
> or a non-initialized {{PigStats}} depending on the order of the tests. If
> it's a previous {{PigStats}}, then {{PigMain}} will write the Hadoop Job ID
> to a new file for the {{testPig_withNullExternalID}} test, which will then
> fail because it's not expecting this file.
> Unfortunately, there isn't a clean way to reset {{PigStats}}. Pig v 0.9 has
> a {{set}} method which we can pass {{null}} to in order to reset the
> {{PigStats}}. However, this was changed in Pig v 0.13 to the {{start}}
> method. In either case, they're both package private, so we have to
> usereflection to find the existing method for whichever version of Pig we're
> using and make it public. We should do this after every Pig test, even if
> they don't all need it just to make sure that we're starting with a fresh Pig
> each time as unit tests are supposed to start fresh anyway.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
