[
https://issues.apache.org/jira/browse/SPARK-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14242030#comment-14242030
]
Michael Armbrust commented on SPARK-4814:
-----------------------------------------
Hive throws a lot of warning here for some reason even when reading data from
their own test files. One option here would be to just override the
configuration only for the hive subproject.
{code}
// Hive throws assertion errors for tests that pass.
javaOptions in Test := (javaOptions in Test).value.filterNot(_ == "-ea"),
{code}
> Enable assertions in SBT, Maven tests / AssertionError from Hive's
> LazyBinaryInteger
> ------------------------------------------------------------------------------------
>
> Key: SPARK-4814
> URL: https://issues.apache.org/jira/browse/SPARK-4814
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
> Affects Versions: 1.1.0
> Reporter: Sean Owen
>
> Follow up to SPARK-4159, wherein we noticed that Java tests weren't running
> in Maven, in part because a Java test actually fails with {{AssertionError}}.
> That code/test was fixed in SPARK-4850.
> The reason it wasn't caught by SBT tests was that they don't run with
> assertions on, and Maven's surefire does.
> Turning on assertions in the SBT build is trivial, adding one line:
> {code}
> javaOptions in Test += "-ea",
> {code}
> This reveals a test failure in Scala test suites though:
> {code}
> [info] - alter_merge_2 *** FAILED *** (1 second, 305 milliseconds)
> [info] Failed to execute query using catalyst:
> [info] Error: Job aborted due to stage failure: Task 1 in stage 551.0
> failed 1 times, most recent failure: Lost task 1.0 in stage 551.0 (TID 1532,
> localhost): java.lang.AssertionError
> [info] at
> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryInteger.init(LazyBinaryInteger.java:51)
> [info] at
> org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase$FieldInfo.uncheckedGetField(ColumnarStructBase.java:110)
> [info] at
> org.apache.hadoop.hive.serde2.columnar.ColumnarStructBase.getField(ColumnarStructBase.java:171)
> [info] at
> org.apache.hadoop.hive.serde2.objectinspector.ColumnarStructObjectInspector.getStructFieldData(ColumnarStructObjectInspector.java:166)
> [info] at
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$1.apply(TableReader.scala:318)
> [info] at
> org.apache.spark.sql.hive.HadoopTableReader$$anonfun$fillObject$1.apply(TableReader.scala:314)
> [info] at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
> [info] at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$6.apply(Aggregate.scala:132)
> [info] at
> org.apache.spark.sql.execution.Aggregate$$anonfun$execute$1$$anonfun$6.apply(Aggregate.scala:128)
> [info] at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:615)
> [info] at org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:615)
> [info] at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> [info] at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
> [info] at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
> [info] at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> [info] at
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:264)
> [info] at org.apache.spark.rdd.RDD.iterator(RDD.scala:231)
> [info] at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> [info] at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> [info] at org.apache.spark.scheduler.Task.run(Task.scala:56)
> [info] at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:195)
> [info] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [info] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [info] at java.lang.Thread.run(Thread.java:745)
> {code}
> The items for this JIRA are therefore:
> - Enable assertions in SBT
> - Fix this failure
> - Figure out why Maven scalatest didn't trigger it - may need assertions
> explicitly turned on too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
