Github user zhengruifeng commented on a diff in the pull request:
https://github.com/apache/spark/pull/19186#discussion_r138237760
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala
---
@@ -483,24 +488,17 @@ class LogisticRegression @Since("1.2.0") (
this
}
- override protected[spark] def train(dataset: Dataset[_]):
LogisticRegressionModel = {
- val handlePersistence = dataset.rdd.getStorageLevel ==
StorageLevel.NONE
- train(dataset, handlePersistence)
- }
-
- protected[spark] def train(
- dataset: Dataset[_],
- handlePersistence: Boolean): LogisticRegressionModel = {
+ protected[spark] def train(dataset: Dataset[_]): LogisticRegressionModel
= {
val w = if (!isDefined(weightCol) || $(weightCol).isEmpty) lit(1.0)
else col($(weightCol))
val instances: RDD[Instance] =
dataset.select(col($(labelCol)), w, col($(featuresCol))).rdd.map {
case Row(label: Double, weight: Double, features: Vector) =>
Instance(label, weight, features)
}
- if (handlePersistence) instances.persist(StorageLevel.MEMORY_AND_DISK)
+ if ($(handlePersistence))
instances.persist(StorageLevel.MEMORY_AND_DISK)
--- End diff --
+1. I supposed that it's up to the users to check the `storageLevel` to
avoid double caching. But I now approve to check in the algs, and it may be
better to log a warning if the dataset is already cached and the
`handlePersistence` is set `true`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
