Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/7564#issuecomment-123409358
In this latest round of tests, it looks like HiveThriftBinaryServerSuite
failed a test:
```
[info] - test multiple session *** FAILED *** (2 seconds, 488 milliseconds)
[info] java.sql.SQLException:
org.apache.spark.sql.catalyst.errors.package$TreeNodeException: sort, tree:
[info] UnsafeExternalSort [KEY#270 DESC], true, 0
[info] ConvertToSafe
[info] Exchange rangepartitioning(KEY#270 DESC)
[info] UnsafeExternalSort [key#270 DESC], true, 0
[info] Exchange rangepartitioning(key#270 DESC)
[info] HiveTableScan [key#270], (MetastoreRelation default, test_map,
None)
[info] at
org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:275)
[info] at
org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:355)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite$$anonfun$8$$anonfun$apply$mcV$sp$13.apply(HiveThriftServer2Suites.scala:327)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite$$anonfun$8$$anonfun$apply$mcV$sp$13.apply(HiveThriftServer2Suites.scala:324)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest$$anonfun$withMultipleConnectionJdbcStatement$1.apply(HiveThriftServer2Suites.scala:438)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest$$anonfun$withMultipleConnectionJdbcStatement$1.apply(HiveThriftServer2Suites.scala:438)
[info] at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
[info] at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftJdbcTest.withMultipleConnectionJdbcStatement(HiveThriftServer2Suites.scala:438)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite$$anonfun$8.apply$mcV$sp(HiveThriftServer2Suites.scala:209)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite$$anonfun$8.apply(HiveThriftServer2Suites.scala:204)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftBinaryServerSuite$$anonfun$8.apply(HiveThriftServer2Suites.scala:204)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
[info] at
org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42)
[info] at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
[info] at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
[info] at scala.collection.immutable.List.foreach(List.scala:318)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info] at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
[info] at
org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
[info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
[info] at org.scalatest.Suite$class.run(Suite.scala:1424)
[info] at
org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
[info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2Test.org$scalatest$BeforeAndAfterAll$$super$run(HiveThriftServer2Suites.scala:450)
[info] at
org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
[info] at
org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
[info] at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2Test.run(HiveThriftServer2Suites.scala:450)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:294)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:284)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[info] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[info] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[info] at java.lang.Thread.run(Thread.java:745)
```
It looks like there may also be a planner bug that occurs when `unsafe` is
enabled while `codegen` is disabled:
```
org.apache.spark.sql.hive.execution.HiveWindowFunctionQueryWithoutCodeGenSuite.
testMultipleRangeWindows
Failed to execute query using catalyst: Error: assertion failed:
UnsafeExternalSort requires code generation to be enabled
java.lang.AssertionError: assertion failed: UnsafeExternalSort requires code
generation to be enabled at scala.Predef$.assert(Predef.scala:179) at
org.apache.spark.sql.execution.UnsafeExternalSort$$anonfun$doExecute$7.apply(basicOperators.scala:281)
at
org.apache.spark.sql.execution.UnsafeExternalSort$$anonfun$doExecute$7.apply(basicOperators.scala:280)
at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:48)
at
```
This is probably easy to fix by adding configuration validation which does
not allow `unsafe` to be enabled unless codegen is also enabled, then updating
the relevant non-code-gen suites to also disable unsafe. We can also modify
the planner to guard against this by also checking whether codegen is enabled
when figuring out whether an unsafe sort operator can be used.
Finally, it looks like a number of the HiveCompatibilitySuite tests are
failing due to the minimum buffer size issue, which I'll try to fix today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
