Github user WeichenXu123 commented on a diff in the pull request:
https://github.com/apache/spark/pull/20686#discussion_r171762299
--- Diff:
mllib/src/test/scala/org/apache/spark/ml/feature/SQLTransformerSuite.scala ---
@@ -63,13 +68,17 @@ class SQLTransformerSuite
}
test("SPARK-22538: SQLTransformer should not unpersist given dataset") {
- val df = spark.range(10)
+ val df = spark.range(10).toDF()
df.cache()
df.count()
assert(df.storageLevel != StorageLevel.NONE)
- new SQLTransformer()
+ val sqlTrans = new SQLTransformer()
.setStatement("SELECT id + 1 AS id1 FROM __THIS__")
- .transform(df)
- assert(df.storageLevel != StorageLevel.NONE)
+ testTransformerByGlobalCheckFunc[Long](
+ df,
+ sqlTrans,
+ "id1") { rows =>
+ assert(df.storageLevel != StorageLevel.NONE)
--- End diff --
Move `assert(df.storageLevel != StorageLevel.NONE)` to here seems
meaningless, because you do not use `rows` parameter.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
