[
https://issues.apache.org/jira/browse/SPARK-22925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-22925.
----------------------------------
Resolution: Incomplete
> ml model persistence creates a lot of small files
> -------------------------------------------------
>
> Key: SPARK-22925
> URL: https://issues.apache.org/jira/browse/SPARK-22925
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 2.1.2, 2.2.1, 2.3.0
> Reporter: Felix Cheung
> Priority: Major
> Labels: bulk-closed
>
> Today in when calling model.save(), some ML models we do makeRDD(data, 1) or
> repartition(1) but in some other models we don't.
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala#L60
> In the former case issues such as SPARK-19294 have been reported for making
> very large single file.
> Whereas in the latter case, models such as RandomForestModel could create
> hundreds or thousands of very small files which is also unmanageable. Looking
> into this, there is no simple way to set/change spark.default.parallelism
> (which would be pick up by sc.parallelize) while the app is running since
> SparkConf seems to be copied/cached by the backend without a way to update
> them.
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala#L443
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala#L155
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala#L135
> It seems we need to have a way to make numSlice settable on a per-use basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
