Github user smurching commented on a diff in the pull request:
https://github.com/apache/spark/pull/19186#discussion_r138137893
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/classification/OneVsRest.scala ---
@@ -163,9 +165,7 @@ final class OneVsRestModel private[ml] (
val initUDF = udf { () => Map[Int, Double]() }
val newDataset = dataset.withColumn(accColName, initUDF())
- // persist if underlying dataset is not persistent.
- val handlePersistence = dataset.rdd.getStorageLevel ==
StorageLevel.NONE
- if (handlePersistence) {
+ if ($(handlePersistence)) {
--- End diff --
See comment above, we should also check that `dataset.storageLevel ==
StorageLevel.NONE` before caching `newDataset`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
