Hi, cos:
I met the same problem before. Can you try below in your Bigtop pig
maven execution .pom ? It will actually resolve other problem as well (for
example, you may see Pig cannot get kerberos principle in secure YARN
cluster)
<dependencies>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pigsmoke</artifactId>
<version>${pig-smoke.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-jobclient</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-hs</artifactId>
<version>${hadoop.version}</version>
</dependency>
Johnny
On Fri, Mar 29, 2013 at 2:54 PM, Konstantin Boudnik <c...@apache.org> wrote:
> I certainly agree with you. I have managed to narrow it down to the
> difference
> between pig-0.10.0-withouthadoop.jar and maven pig-0.10.0.jar.
>
> Basically, I can run:
>
> java -cp \
> /etc/hadoop/conf:\
> /etc/pig/conf:\
> /root/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:\
>
> /root/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:\
>
> /root/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:\
> /root/.m2/repository/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1.jar:\
> /root/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:\
> /usr/lib/hadoop/lib/protobuf-java-2.4.0a.jar:\
> /root/.m2/repository/org/apache/avro/avro/1.5.3/avro-1.5.3.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-common/2.0.3.1-SNAPSHOT/hadoop-common-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-auth/2.0.3.1-SNAPSHOT/hadoop-auth-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.0.3.1-SNAPSHOT/hadoop-hdfs-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.0.3.1-SNAPSHOT/hadoop-yarn-api-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.0.3.1-SNAPSHOT/hadoop-yarn-client-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.0.3.1-SNAPSHOT/hadoop-yarn-common-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-core-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-common-2.0.3.1-SNAPSHOT.jar:\
>
> /root/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.0.3.1-SNAPSHOT/hadoop-mapreduce-client-jobclient-2.0.3.1-SNAPSHOT.jar:\
> /usr/lib/pig/pig-0.10.0-withouthadoop.jar:\
> /root/.m2/repository/com/google/guava/guava/11.0/guava-11.0.jar:\
> /root/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:\
> /root/.m2/repository/org/antlr/antlr-runtime/3.4/antlr-runtime-3.4.jar:\
>
> /root/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:\
>
> /root/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:\
>
> /root/.m2/repository/org/antlr/stringtemplate/3.2.1/stringtemplate-3.2.1.jar:\
> /usr/lib/pig/lib/automaton.jar:\
> build/ivy/lib/Pig/junit-4.5.jar:\
> build/pig-0.10.0-SNAPSHOT-smoketests.jar:\
>
> /root/.m2/repository/org/apache/pig/pigunit/0.10.0-SNAPSHOT/pigunit-0.10.0-SNAPSHOT.jar
> \
> -Dpigunit.exectype.cluster=true \
> org.junit.runner.JUnitCore org.apache.pig.test.pigunit.TestPigTest
>
> But once
> /usr/lib/pig/pig-0.10.0-withouthadoop.jar
> is replaced with
> /root/.m2/repository/org/apache/pig/pig/0.10.0/pig-0.10.0.jar
> all hell breaks loose and submission doesn't happen anymore ;( And as
> before -
> no error messages nor exceptions appear anywhere on the surface.
>
> Cos
>
> On Fri, Mar 29, 2013 at 11:52AM, Dmitriy Ryaboy wrote:
> > I missed the part where you are running it from JUnit. Still suspect a
> > classpath issue :). Can you dump your classpath as seen by JUnit, and
> what
> > happens when you run "-secretDebugCmd" and see if something is missing?
> >
> > -D
> >
> >
> > On Fri, Mar 29, 2013 at 10:38 AM, Konstantin Boudnik <c...@apache.org>
> wrote:
> >
> > > There's no problem with Pig itself. If I use Pig CLI I can run some
> > > PigLatin
> > > scripts and they go all the way to the completion.
> > >
> > > The problem arises while running TestPigTest _programmatically_ e.g.
> using
> > > JUnit runner. Is there any secret debug options in this case? I've
> looked
> > > through the code but didn't spot anything immediately.
> > >
> > > I will give "-secretDebugCmd" a try anyway - it might give me some
> ideas...
> > >
> > > Appreciate the help!
> > > Cos
> > >
> > > On Fri, Mar 29, 2013 at 09:08AM, Dmitriy Ryaboy wrote:
> > > > Can you run `pig -secretDebugCmd` and look at your paths? I suspect
> > > there's
> > > > something going on there, like not having the right configs or hadoop
> > > jars.
> > > >
> > > >
> > > > On Thu, Mar 28, 2013 at 11:32 PM, Konstantin Boudnik <c...@apache.org
> >
> > > wrote:
> > > >
> > > > > Guys,
> > > > >
> > > > > I am trying to run TestPigTest against a fanctional Hadoop 2.0 YARN
> > > > > cluster -
> > > > > I can run MR jobs using Hive or plain MR examples. However, it is
> going
> > > > > somewhat unsuccessful. While using pig "far" jar file - e.g. with
> > > > > everything
> > > > > repacked inside of it - I seeminly can run TestPigTest against a
> real
> > > > > cluster.
> > > > >
> > > > > However, while using pig-withouthadoop.jar and adding all needed
> > > > > dependencies
> > > > > to the classpath, Pig is failing to communicate with YARN nor
> submit
> > > any MR
> > > > > jobs. There's no exceptions, not error messages anywhere in the
> logs.
> > > Have
> > > > > anyone seen anything similar?
> > > > >
> > > > > Any input is highly appreciated!
> > > > >
> > > > > --
> > > > > With regards,
> > > > > Cos
> > > > >
> > > > >
> > >
>
