Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/7462#issuecomment-122759346
I pushed updates to the code based on the comments above. I wrote an
expressions test, but can't figure out why its failing. I get the following
stacktrace:
```
[info] CollectionFunctionsSuite:
[info] - Array and Map Size *** FAILED *** (958 milliseconds)
[info] Exception evaluating size([I@7e14fccd)
(ExpressionEvalHelper.scala:69)
[info] org.scalatest.exceptions.TestFailedException:
[info] at
org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:496)
[info] at
org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555)
[info] at org.scalatest.Assertions$class.fail(Assertions.scala:1348)
[info] at org.scalatest.FunSuite.fail(FunSuite.scala:1555)
[info] at
org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper$class.checkEvaluationWithoutCodegen(ExpressionEvalHelper.scala:69)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite.checkEvaluationWithoutCodegen(CollectionFunctionsSuite.scala:24)
[info] at
org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper$class.checkEvaluation(ExpressionEvalHelper.scala:43)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite.checkEvaluation(CollectionFunctionsSuite.scala:24)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply$mcV$sp(CollectionFunctionsSuite.scala:31)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply(CollectionFunctionsSuite.scala:26)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply(CollectionFunctionsSuite.scala:26)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
[info] at
org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42)
[info] at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
[info] at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
[info] at scala.collection.immutable.List.foreach(List.scala:318)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info] at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
[info] at
org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
[info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
[info] at org.scalatest.Suite$class.run(Suite.scala:1424)
[info] at
org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
[info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
[info] at org.scalatest.FunSuite.run(FunSuite.scala:1555)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:294)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:284)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[info] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[info] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[info] at java.lang.Thread.run(Thread.java:745)
[info] Cause: java.lang.ClassCastException: [I cannot be cast to
scala.collection.Seq
[info] at
org.apache.spark.sql.catalyst.expressions.Size.nullSafeEval(collectionOperations.scala:30)
[info] at
org.apache.spark.sql.catalyst.expressions.Size.nullSafeEval(collectionOperations.scala:25)
[info] at
org.apache.spark.sql.catalyst.expressions.UnaryExpression.eval(Expression.scala:219)
[info] at
org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper$class.evaluate(ExpressionEvalHelper.scala:62)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite.evaluate(CollectionFunctionsSuite.scala:24)
[info] at
org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper$class.checkEvaluationWithoutCodegen(ExpressionEvalHelper.scala:69)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite.checkEvaluationWithoutCodegen(CollectionFunctionsSuite.scala:24)
[info] at
org.apache.spark.sql.catalyst.expressions.ExpressionEvalHelper$class.checkEvaluation(ExpressionEvalHelper.scala:43)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite.checkEvaluation(CollectionFunctionsSuite.scala:24)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply$mcV$sp(CollectionFunctionsSuite.scala:31)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply(CollectionFunctionsSuite.scala:26)
[info] at
org.apache.spark.sql.catalyst.expressions.CollectionFunctionsSuite$$anonfun$1.apply(CollectionFunctionsSuite.scala:26)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] at org.scalatest.Transformer.apply(Transformer.scala:22)
[info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[info] at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
[info] at
org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42)
[info] at
org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
[info] at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
[info] at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
[info] at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
[info] at
org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
[info] at
org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
[info] at scala.collection.immutable.List.foreach(List.scala:318)
[info] at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
[info] at
org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
[info] at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
[info] at
org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
[info] at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
[info] at org.scalatest.Suite$class.run(Suite.scala:1424)
[info] at
org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
[info] at
org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
[info] at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
[info] at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
[info] at org.scalatest.FunSuite.run(FunSuite.scala:1555)
[info] at
org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)
[info] at
org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:294)
[info] at sbt.ForkMain$Run$2.call(ForkMain.java:284)
[info] at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[info] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[info] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[info] at java.lang.Thread.run(Thread.java:745)
```
What I gather is basically Scala can't convert a class of type `I` (don't
know what type this is, perhaps type erasure makes it no possible to know?) to
`Seq` in `collectionOperations.scala:30`. I am not sure why this is.
What is the difference between tests in dataframes and tests in expressions
(in terms of codepath)?
Other questions from changing code:
1. Difference between `eval` and `nullSafeEval`?
2. Quick summary of types to know for Spark SQL in terms of types of
expressions (Unary, Binary, ExpectsInputTypes).
3. Does `Literal` represent Arrays/Maps, or only primitive types? Looks
like you can either define your own datatype or have scala attempt to infer it
with pattern matching.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
