Hiya! It works for me if I package on stat1002. I was going to help out on https://gerrit.wikimedia.org/r/#/c/345863/ but that's not urgent.
On Fri, Mar 31, 2017 at 2:22 PM, Dan Andreescu <[email protected]> wrote: > Mikhail, I know Joseph was working with that job whose test is failing, > I'll cc him personally to see if he has any idea. Does this work if you do > it on the cluster? Since it's the weekend, you probably won't get a > response until Monday, let me know if there's anything urgent and we can > try to find a work-around. > > On Fri, Mar 31, 2017 at 4:45 PM, Mikhail Popov <[email protected]> > wrote: > >> Hello! Has anyone experienced issues `mvn package`-ing >> analytics/refinery/source on a local machine? >> >> Wikimedia Analytics Refinery Jobs fails for me as of "Add mediawiki >> history spark jobs to refinery-job" (https://gerrit.wikimedia.org/ >> r/#/c/325312/) >> >> Here's my `mvn --version`: >> >> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >> 2015-11-10T08:41:47-08:00) >> Maven home: /usr/local/Cellar/maven/3.3.9/libexec >> Java version: 1.8.0_121, vendor: Oracle Corporation >> Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/ >> Home/jre >> Default locale: en_US, platform encoding: UTF-8 >> OS name: "mac os x", version: "10.12.4", arch: "x86_64", family: "mac" >> >> When I set HEAD to the commit prior to that everything succeeds. Any >> commit after that one makes Jobs tests fail with warnings and errors like: >> >> [INFO] Checking for multiple versions of scala >> [WARNING] Expected all dependencies to require Scala version: 2.10.4 >> [WARNING] com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4 >> [WARNING] org.spark-project.akka:akka-actor_2.10:2.2.3-shaded-protobuf >> requires scala version: 2.10.4 >> [WARNING] org.spark-project.akka:akka-remote_2.10:2.2.3-shaded-protobuf >> requires scala version: 2.10.4 >> [WARNING] org.spark-project.akka:akka-slf4j_2.10:2.2.3-shaded-protobuf >> requires scala version: 2.10.4 >> [WARNING] org.apache.spark:spark-core_2.10:1.6.0-cdh5.10.0 requires >> scala version: 2.10.4 >> [WARNING] org.json4s:json4s-jackson_2.10:3.2.10 requires scala version: >> 2.10.4 >> [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: >> 2.10.4 >> [WARNING] org.json4s:json4s-ast_2.10:3.2.10 requires scala version: >> 2.10.4 >> [WARNING] org.json4s:json4s-core_2.10:3.2.10 requires scala version: >> 2.10.0 >> [WARNING] Multiple versions of scala libraries detected! >> >> TestDenormalizedRevisionsBuilder: >> 17/03/31 13:38:41 WARN NativeCodeLoader: Unable to load native-hadoop >> library for your platform... using builtin-java classes where applicable >> >> populateDeleteTime >> java.lang.reflect.InvocationTargetException >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >> ssorImpl.java:62) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:498) >> at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoade >> r.java:317) >> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219) >> at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44) >> at org.apache.spark.io.SnappyCompressionCodec$.liftedTree1$1(Co >> mpressionCodec.scala:169) >> at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark >> $io$SnappyCompressionCodec$$version$lzycompute(Compression >> Codec.scala:168) >> at org.apache.spark.io.SnappyCompressionCodec$.org$apache$spark >> $io$SnappyCompressionCodec$$version(CompressionCodec.scala:168) >> at org.apache.spark.io.SnappyCompressionCodec.<init>(Compressio >> nCodec.scala:152) >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >> at sun.reflect.NativeConstructorAccessorImpl.newInstance(Native >> ConstructorAccessorImpl.java:62) >> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(De >> legatingConstructorAccessorImpl.java:45) >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) >> at org.apache.spark.io.CompressionCodec$.createCodec(Compressio >> nCodec.scala:72) >> at org.apache.spark.io.CompressionCodec$.createCodec(Compressio >> nCodec.scala:65) >> at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark >> $broadcast$TorrentBroadcast$$setConf(TorrentBroadcast.scala:73) >> at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBr >> oadcast.scala:80) >> at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadc >> ast(TorrentBroadcastFactory.scala:34) >> at org.apache.spark.broadcast.BroadcastManager.newBroadcast(Bro >> adcastManager.scala:63) >> at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1334) >> at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(D >> AGScheduler.scala:1006) >> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$sch >> eduler$DAGScheduler$$submitStage(DAGScheduler.scala:921) >> at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$ >> spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:924) >> at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$ >> spark$scheduler$DAGScheduler$$submitStage$4.apply(DAGScheduler.scala:923) >> at scala.collection.immutable.List.foreach(List.scala:318) >> at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$sch >> eduler$DAGScheduler$$submitStage(DAGScheduler.scala:923) >> at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(D >> AGScheduler.scala:861) >> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOn >> Receive(DAGScheduler.scala:1611) >> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onRe >> ceive(DAGScheduler.scala:1603) >> at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onRe >> ceive(DAGScheduler.scala:1592) >> at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) >> Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in >> java.library.path >> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) >> at java.lang.Runtime.loadLibrary0(Runtime.java:870) >> at java.lang.System.loadLibrary(System.java:1122) >> at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativ >> eLoader.java:52) >> ... 33 more >> - should put max rev ts when no page state match *** FAILED *** >> org.apache.spark.SparkException: Job aborted due to stage failure: >> Task serialization failed: java.lang.reflect.InvocationTargetException >> >> _______________________________________________ >> Analytics mailing list >> [email protected] >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> > > _______________________________________________ > Analytics mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/analytics > >
_______________________________________________ Analytics mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/analytics
