Filed https://issues.apache.org/jira/browse/BIGTOP-1727 and uploaded a patch.
Thanks, Youngwoo On Fri, Mar 6, 2015 at 9:38 AM, 김영우 <[email protected]> wrote: > Gotcha. > > BIGTOP-1716 causes the build failure. I'm looking into this. > > Sorry for your inconvenience, I'll upload a patch. > > Thanks, > Youngwoo > > On Fri, Mar 6, 2015 at 8:05 AM, jay vyas <[email protected]> > wrote: > >> Hi jonathan ! >> >> I did indeed build and test spark 1.2.1 in BIGTOP-1648 : And actually >> during the review i pasted the text output : Seemed to work nicely : >> https://issues.apache.org/jira/browse/BIGTOP-1648 >> >> Lets follow up on this here >> https://issues.apache.org/jira/browse/BIGTOP-1726, where we can retest >> everything. Its quite easy to retest will leave some guidance directions >> there if you want to try it out. >> >> >> On Thu, Mar 5, 2015 at 5:04 PM, Kelly, Jonathan <[email protected]> >> wrote: >> >>> As I said below, I don't think this could be a BigTop issue, but has >>> anybody from the BigTop community seen anything like this? >>> >>> Thanks, >>> Jonathan Kelly >>> >>> >>> >>> >>> On 3/5/15, 1:34 PM, "Kelly, Jonathan" <[email protected]> wrote: >>> >>> >That's probably a good thing to have, so I'll add it, but unfortunately >>> it >>> >did not help this issue. It looks like the hadoop-2.4 profile only sets >>> >these properties, which don't seem like they would affect anything >>> related >>> >to Netty: >>> > >>> > <properties> >>> > <hadoop.version>2.4.0</hadoop.version> >>> > <protobuf.version>2.5.0</protobuf.version> >>> > <jets3t.version>0.9.0</jets3t.version> >>> > <commons.math3.version>3.1.1</commons.math3.version> >>> > <avro.mapred.classifier>hadoop2</avro.mapred.classifier> >>> > </properties> >>> > >>> > >>> >Thanks, >>> >Jonathan Kelly >>> > >>> > >>> > >>> > >>> >On 3/5/15, 1:09 PM, "Patrick Wendell" <[email protected]> wrote: >>> > >>> >>You may need to add the -Phadoop-2.4 profile. When building or release >>> >>packages for Hadoop 2.4 we use the following flags: >>> >> >>> >>-Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn >>> >> >>> >>- Patrick >>> >> >>> >>On Thu, Mar 5, 2015 at 12:47 PM, Kelly, Jonathan <[email protected]> >>> >>wrote: >>> >>> I confirmed that this has nothing to do with BigTop by running the >>> same >>> >>>mvn >>> >>> command directly in a fresh clone of the Spark package at the v1.2.1 >>> >>>tag. I >>> >>> got the same exact error. >>> >>> >>> >>> >>> >>>~ Jonathan Kelly >>> >>> >>> >>> >>> >>> From: <Kelly>, Jonathan Kelly <[email protected]> >>> >>> Date: Thursday, March 5, 2015 at 10:39 AM >>> >>> To: "[email protected]" <[email protected]> >>> >>> Subject: Spark v1.2.1 failing under BigTop build in External Flume >>> Sink >>> >>>(due >>> >>> to missing Netty library) >>> >>> >>> >>> I'm running into an issue building Spark v1.2.1 (as well as the >>> latest >>> >>>in >>> >>> branch-1.2 and v1.3.0-rc2 and the latest in branch-1.3) with BigTop >>> >>>(v0.9, >>> >>> which is not quite released yet). The build fails in the External >>> >>>Flume >>> >>> Sink subproject with the following error: >>> >>> >>> >>> [INFO] Compiling 5 Scala sources and 3 Java sources to >>> >>> >>> >>> >>>/workspace/workspace/bigtop.spark-rpm/build/spark/rpm/BUILD/spark-1.3.0/ >>> >>>e >>> >>>xternal/flume-sink/target/scala-2.10/classes... >>> >>> [WARNING] Class org.jboss.netty.channel.ChannelFactory not found - >>> >>> continuing with a stub. >>> >>> [ERROR] error while loading NettyServer, class file >>> >>> >>> >>> >>>'/home/ec2-user/.m2/repository/org/apache/avro/avro-ipc/1.7.6/avro-ipc-1 >>> >>>. >>> >>>7.6.jar(org/apache/avro/ipc/NettyServer.class)' >>> >>> is broken >>> >>> (class java.lang.NullPointerException/null) >>> >>> [WARNING] one warning found >>> >>> [ERROR] one error found >>> >>> >>> >>> It seems like what is happening is that the Netty library is missing >>> at >>> >>> build time, which happens because it is explicitly excluded in the >>> >>>pom.xml >>> >>> (see >>> >>> >>> >>> >>> https://github.com/apache/spark/blob/v1.2.1/external/flume-sink/pom.xml# >>> >>>L >>> >>>42). >>> >>> I attempted removing the exclusions and the explicit re-add for the >>> >>>test >>> >>> scope on lines 77-88, and that allowed the build to succeed, though I >>> >>>don't >>> >>> know if that will cause problems at runtime. I don't have any >>> >>>experience >>> >>> with the Flume Sink, so I don't really know how to test it. (And, to >>> >>>be >>> >>> clear, I'm not necessarily trying to get the Flume Sink to work-- I >>> >>>just >>> >>> want the project to build successfully, though of course I'd still >>> want >>> >>>the >>> >>> Flume Sink to work for whomever does need it.) >>> >>> >>> >>> Does anybody have any idea what's going on here? Here is the command >>> >>>BigTop >>> >>> is running to build Spark: >>> >>> >>> >>> mvn -Pbigtop-dist -Pyarn -Phive -Phive-thriftserver -Pkinesis-asl >>> >>> -Divy.home=/home/ec2-user/.ivy2 -Dsbt.ivy.home=/home/ec2-user/.ivy2 >>> >>> -Duser.home=/home/ec2-user -Drepo.maven.org= >>> >>> -Dreactor.repo=file:///home/ec2-user/.m2/repository >>> >>> -Dhadoop.version=2.4.0-amzn-3-SNAPSHOT >>> >>>-Dyarn.version=2.4.0-amzn-3-SNAPSHOT >>> >>> -Dprotobuf.version=2.5.0 -Dscala.version=2.10.3 >>> >>>-Dscala.binary.version=2.10 >>> >>> -DskipTests -DrecompileMode=all install >>> >>> >>> >>> As I mentioned above, if I switch to the latest in branch-1.2, to >>> >>> v1.3.0-rc2, or to the latest in branch-1.3, I get the same exact >>> error. >>> >>> I >>> >>> was not getting the error with Spark v1.1.0, though there weren't any >>> >>> changes to the external/flume-sink/pom.xml between v1.1.0 and v1.2.1. >>> >>> >>> >>> >>> >>> ~ Jonathan Kelly >>> > >>> >>> >> >> >> -- >> jay vyas >> > >
