[
https://issues.apache.org/jira/browse/SPARK-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14246024#comment-14246024
]
Michael Armbrust commented on SPARK-4814:
-----------------------------------------
Either way, I don't think we should block turning on assertions for the rest of
Spark.
> Enable assertions in SBT, Maven tests / AssertionError from Hive's
> LazyBinaryInteger
> ------------------------------------------------------------------------------------
>
> Key: SPARK-4814
> URL: https://issues.apache.org/jira/browse/SPARK-4814
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 1.1.0
> Reporter: Sean Owen
>
> Follow up to SPARK-4159, wherein we noticed that Java tests weren't running
> in Maven, in part because a Java test actually fails with {{AssertionError}}.
> That code/test was fixed in SPARK-4850.
> The reason it wasn't caught by SBT tests was that they don't run with
> assertions on, and Maven's surefire does.
> Turning on assertions in the SBT build is trivial, adding one line:
> {code}
> javaOptions in Test += "-ea",
> {code}
> This reveals a test failure in Scala test suites though:
> {code}
> [info] - alter_merge_2 *** FAILED *** (1 second, 305 milliseconds)
> [info] Failed to execute query using catalyst:
> [info] Error: Job aborted due to stage failure: Task 1 in stage 551.0
> failed 1 times, most recent failure: Lost task 1.0 in stage 551.0 (TID 1532,
> localhost): java.lang.AssertionError
> [info] at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryInteger.init(LazyBinaryInteger.java:51)
> [info] at
> org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:110)
> [info] at
> org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getField(ColumnarStructBase.java:171)
> [info] at
> org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldData(ColumnarStructObjectInspector.java:166)
> [info] at
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$1.apply(TableReader.scala:318)
> [info] at
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$1.apply(TableReader.scala:314)
> [info] at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> [info] at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$6.apply(Aggregate.scala:132)
> [info] at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$6.apply(Aggregate.scala:128)
> [info] at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:615)
> [info] at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:615)
> [info] at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> [info] at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
> [info] at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
> [info] at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> [info] at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
> [info] at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
> [info] at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> [info] at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> [info] at org.apache.spark.scheduler.Task.run(Task.scala:56)
> [info] at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:195)
> [info] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [info] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [info] at java.lang.Thread.run(Thread.java:745)
> {code}
> The items for this JIRA are therefore:
> - Enable assertions in SBT
> - Fix this failure
> - Figure out why Maven scalatest didn't trigger it - may need assertions
> explicitly turned on too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
