[spark] branch master updated: [SPARK-26526][SQL][TEST] Fix invalid test case about non-deterministic expression

[wenchen]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f65dc95  [SPARK-26526][SQL][TEST] Fix invalid test case about 
non-deterministic expression
f65dc95 is described below

commit f65dc9593ee4b84343fea04fdcace14096788be8
Author: Liu,Linhong <liulinh...@baidu.com>
AuthorDate: Fri Jan 4 10:51:33 2019 +0800

    [SPARK-26526][SQL][TEST] Fix invalid test case about non-deterministic 
expression
    
    ## What changes were proposed in this pull request?
    
    Test case in SPARK-10316 is used to make sure non-deterministic `Filter` 
won't be pushed through `Project`
    But in current code base this test case can't cover this purpose.
    Change LogicalRDD to HadoopFsRelation can fix this issue.
    
    ## How was this patch tested?
    
    Modified test pass.
    
    Closes #23440 from LinhongLiu/fix-test.
    
    Authored-by: Liu,Linhong <liulinh...@baidu.com>
    Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
 .../src/test/scala/org/apache/spark/sql/DataFrameSuite.scala  | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index b51c51e..3082e0b 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -1398,11 +1398,14 @@ class DataFrameSuite extends QueryTest with 
SharedSQLContext {
   }
 
   test("SPARK-10316: respect non-deterministic expressions in 
PhysicalOperation") {
-    val input = spark.read.json((1 to 10).map(i => s"""{"id": $i}""").toDS())
+    withTempDir { dir =>
+      (1 to 
10).toDF("id").write.mode(SaveMode.Overwrite).json(dir.getCanonicalPath)
+      val input = spark.read.json(dir.getCanonicalPath)
 
-    val df = input.select($"id", rand(0).as('r))
-    df.as("a").join(df.filter($"r" < 0.5).as("b"), $"a.id" === 
$"b.id").collect().foreach { row =>
-      assert(row.getDouble(1) - row.getDouble(3) === 0.0 +- 0.001)
+      val df = input.select($"id", rand(0).as('r))
+      df.as("a").join(df.filter($"r" < 0.5).as("b"), $"a.id" === 
$"b.id").collect().foreach { row =>
+        assert(row.getDouble(1) - row.getDouble(3) === 0.0 +- 0.001)
+      }
     }
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org