Sigs, license, etc are OK. There are no Blockers for 2.0.2, though here are
the 4 issues still open:
SPARK-14387 Enable Hive-1.x ORC compatibility with
spark.sql.hive.convertMetastoreOrc
SPARK-17957 Calling outer join and na.fill(0) and then inner join will miss
rows
SPARK-17981 Incorrectly Set Nullability to False in FilterExec
SPARK-18160 spark.files & spark.jars should not be passed to driver in yarn
mode
Running with Java 8, -Pyarn -Phive -Phive-thriftserver -Phadoop-2.7 on
Ubuntu 16, I am seeing consistent failures in this test below. I think we
very recently changed this so it could be legitimate. But does anyone else
see something like this? I have seen other failures in this test due to OOM
but my MAVEN_OPTS allows 6g of heap, which ought to be plenty.
- SPARK-18189: Fix serialization issue in KeyValueGroupedDataset *** FAILED
***
isContain was true Interpreter output contained 'Exception':
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.0.2
/_/
Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_102)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
scala> keyValueGrouped:
org.apache.spark.sql.KeyValueGroupedDataset[Int,(Int, Int)] =
org.apache.spark.sql.KeyValueGroupedDataset@70c30f72
scala> mapGroups: org.apache.spark.sql.Dataset[(Int, Int)] = [_1: int,
_2: int]
scala> broadcasted: org.apache.spark.broadcast.Broadcast[Int] =
Broadcast(0)
scala>
scala>
scala> dataset: org.apache.spark.sql.Dataset[Int] = [value: int]
scala> org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in
stage 0.0 (TID 0, localhost):
com.google.common.util.concurrent.ExecutionError:
java.lang.ClassCircularityError:
io/netty/util/internal/__matchers__/org/apache/spark/network/protocol/MessageMatcher
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2261)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:841)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:188)
at
org.apache.spark.sql.catalyst.expressions.codegen.GenerateSafeProjection$.create(GenerateSafeProjection.scala:36)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:825)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:822)
at
org.apache.spark.sql.execution.ObjectOperator$.deserializeRowToObject(objects.scala:137)
at
org.apache.spark.sql.execution.AppendColumnsExec$$anonfun$9.apply(objects.scala:251)
at
org.apache.spark.sql.execution.AppendColumnsExec$$anonfun$9.apply(objects.scala:250)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:803)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCircularityError:
io/netty/util/internal/__matchers__/org/apache/spark/network/protocol/MessageMatcher
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:62)
at
io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:54)
at
io.netty.util.internal.TypeParameterMatcher.get(TypeParameterMatcher.java:42)
at
io.netty.util.internal.TypeParameterMatcher.find(TypeParameterMatcher.java:78)
at
io.netty.handler.codec.MessageToMessageEncoder.<init>(MessageToMessageEncoder.java:60)
at
org.apache.spark.network.protocol.MessageEncoder.<init>(MessageEncoder.java:34)
at
org.apache.spark.network.TransportContext.<init>(TransportContext.java:78)
at
org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:354)
at
org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
at org.apache.spark.repl.ExecutorClassLoader.org
$apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:90)
at
org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57)
at
org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57)
at
org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:161)
at
org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:80)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:62)
at
io.netty.util.internal.JavassistTypeParameterMatcherGenerator.generate(JavassistTypeParameterMatcherGenerator.java:54)
at
io.netty.util.internal.TypeParameterMatcher.get(TypeParameterMatcher.java:42)
at
io.netty.util.internal.TypeParameterMatcher.find(TypeParameterMatcher.java:78)
at
io.netty.handler.codec.MessageToMessageEncoder.<init>(MessageToMessageEncoder.java:60)
at
org.apache.spark.network.protocol.MessageEncoder.<init>(MessageEncoder.java:34)
at
org.apache.spark.network.TransportContext.<init>(TransportContext.java:78)
at
org.apache.spark.rpc.netty.NettyRpcEnv.downloadClient(NettyRpcEnv.scala:354)
at
org.apache.spark.rpc.netty.NettyRpcEnv.openChannel(NettyRpcEnv.scala:324)
at org.apache.spark.repl.ExecutorClassLoader.org
$apache$spark$repl$ExecutorClassLoader$$getClassFileInputStreamFromSparkRPC(ExecutorClassLoader.scala:90)
at
org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57)
at
org.apache.spark.repl.ExecutorClassLoader$$anonfun$1.apply(ExecutorClassLoader.scala:57)
at
org.apache.spark.repl.ExecutorClassLoader.findClassLocally(ExecutorClassLoader.scala:161)
at
org.apache.spark.repl.ExecutorClassLoader.findClass(ExecutorClassLoader.scala:80)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at
org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:34)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.spark.util.ParentClassLoader.loadClass(ParentClassLoader.scala:30)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:78)
at org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:254)
at org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:6893)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5331)
at
org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:5207)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5188)
at org.codehaus.janino.UnitCompiler.access$12600(UnitCompiler.java:185)
at
org.codehaus.janino.UnitCompiler$16.visitReferenceType(UnitCompiler.java:5119)
at org.codehaus.janino.Java$ReferenceType.accept(Java.java:2880)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5159)
at org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:5414)
at org.codehaus.janino.UnitCompiler.access$12400(UnitCompiler.java:185)
at
org.codehaus.janino.UnitCompiler$16.visitArrayType(UnitCompiler.java:5117)
at org.codehaus.janino.Java$ArrayType.accept(Java.java:2954)
at org.codehaus.janino.UnitCompiler.getType(UnitCompiler.java:5159)
at org.codehaus.janino.UnitCompiler.access$16700(UnitCompiler.java:185)
at
org.codehaus.janino.UnitCompiler$31.getParameterTypes2(UnitCompiler.java:8533)
at
org.codehaus.janino.IClass$IInvocable.getParameterTypes(IClass.java:835)
at org.codehaus.janino.IClass$IMethod.getDescriptor2(IClass.java:1063)
at org.codehaus.janino.IClass$IInvocable.getDescriptor(IClass.java:849)
at org.codehaus.janino.IClass.getIMethods(IClass.java:211)
at org.codehaus.janino.IClass.getIMethods(IClass.java:199)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:409)
at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:393)
at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:185)
at
org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:347)
at
org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1139)
at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:354)
at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:322)
at
org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:383)
at
org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:315)
at
org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:233)
at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:192)
at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:84)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:887)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:950)
at
org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:947)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
... 26 more
On Wed, Nov 2, 2016 at 4:52 AM Reynold Xin <r...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 2.0.2. The vote is open until Fri, Nov 4, 2016 at 22:00 PDT and passes if a
> majority of at least 3+1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 2.0.2
> [ ] -1 Do not release this package because ...
>
>
> The tag to be voted on is v2.0.2-rc2
> (a6abe1ee22141931614bf27a4f371c46d8379e33)
>
> This release candidate resolves 84 issues:
> https://s.apache.org/spark-2.0.2-jira
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-2.0.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1210/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-2.0.2-rc2-docs/
>
>
> Q: How can I help test this release?
> A: If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions from 2.0.1.
>
> Q: What justifies a -1 vote for this release?
> A: This is a maintenance release in the 2.0.x series. Bugs already present
> in 2.0.1, missing features, or bugs related to new features will not
> necessarily block this release.
>
> Q: What fix version should I use for patches merging into branch-2.0 from
> now on?
> A: Please mark the fix version as 2.0.3, rather than 2.0.2. If a new RC
> (i.e. RC3) is cut, I will change the fix version of those patches to 2.0.2.
>
