It took a deep debugging session to find the reason why my test failed.

After two hours of tracing step by step through the insides of Pig itself I
noticed it was going to the "cleanupOnFailure" method.
It turns out that the entire job failed in the unit test and the error was
hidden to me.

The error was:
    Unexpected System Error Occured:
java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
    at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:243)
    at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:190)

Sigh ... So it's the Hadoop 1.x vs 2.x problem again.

So why was the message hidden?
Turns out I forgot to put in a log4j.properties file and even though the
junit errors are put on the screen; this error wasn't.

The fix was easy:

Change the pig dependency in the pom.xml from this

     <dependency>
       <groupId>org.apache.pig</groupId>
       <artifactId>pig</artifactId>
       <version>0.13.0</version>
     </dependency>

into this

    <dependency>
      <groupId>org.apache.pig</groupId>
      <artifactId>pig</artifactId>
      <version>0.13.0</version>
      <classifier>h2</classifier>
    </dependency>

Niels Basjes


On Sun, Nov 23, 2014 at 8:29 PM, Niels Basjes <ni...@basjes.nl> wrote:

> I did some additional testing myself and I now have the "smallest
> possible" way to reproduce the effect I see.
>
> I was able to reproduce the problem I'm facing by simply trying to run an
> existing unit test from pig without modification of the code at all.
>
> https://github.com/nielsbasjes/pig-loader-testing
>
> So what I did:
>
> Create a very simple pom.xml
> Download
> https://raw.githubusercontent.com/apache/pig/branch-0.13/test/org/apache/pig/builtin/mock/TestMockStorage.java
> and put it in the right directory.
>
> Now when you run
>
> mvn test
>
>
> I see:
>
> java.lang.RuntimeException: No data for location 'bar'
>>     at org.apache.pig.builtin.mock.Storage$Data.get(Storage.java:327)
>>     at
>> org.apache.pig.builtin.mock.TestMockStorage.testMockStoreAndLoad(TestMockStorage.java:55)
>
>
> and
>
> junit.framework.AssertionFailedError:
>> Expected :{a: chararray,b: chararray}
>> Actual   :null
>
>
> and
>
> java.lang.RuntimeException: No data for location 'output'
>>     at org.apache.pig.builtin.mock.Storage$Data.get(Storage.java:327)
>>     at
>> org.apache.pig.builtin.mock.TestMockStorage.testMockStoreUnion(TestMockStorage.java:106)
>
>
> Any help suggestions to what I'm doing wrong is very welcome.
>
> Niels Basjes
>
>
> On Sun, Nov 23, 2014 at 12:29 AM, Niels Basjes <ni...@basjes.nl> wrote:
>
>> Hi,
>>
>> I have written a Pig Loader and I want to make unit tests for it that
>> actually use it from Pig.
>>
>> It's a Maven project and as a first attempt I created a testing class and
>> tried to create a java based unit test by doing something similar to this:
>>
>> https://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/builtin/mock/TestMockStorage.java
>>
>> I couldn't get it to work so I copied the exact method
>> testMockStoreAndLoad and I still get this error:
>> *java.lang.RuntimeException: No data for location 'bar'*
>>
>> Does anyone have a suggestion on how to proceed?
>>
>> --
>> Best regards
>>
>> Niels Basjes
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>



-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

Reply via email to