[ https://issues.apache.org/jira/browse/SPARK-10078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819964#comment-15819964 ]
Weichen Xu commented on SPARK-10078: ------------------------------------ [~debasish83] But when we implement VF-LBFGS/VF-OWLQN base on spark, we found that many optimizations need to combine spark features and the optimizer algorithm closely, make a abstract interface supporting distributed vector (for example, Vector space operator include dot, add, scale, persist/unpersist operators and so on...) seems not enough. I give two simple problem to show the complexity when considering general interface: 1. Look this VF-OWLQN implementation based on spark: https://github.com/yanboliang/spark-vlbfgs/blob/master/src/main/scala/org/apache/spark/ml/optim/VectorFreeOWLQN.scala We know that OWLQN internal will help compute the pseudo-gradient for L1 reg, look the code function `calculateComponentWithL1`, here when computing pseudo-gradient using RDD, it also use an accumulator(only spark have) to calculate the adjusted fnValue, so that will the abstract interface containing something about `accumulator` in spark ? 2. About persist, unpersist, checkpoint problem in spark. Because of spark lazy computation feature, improper persist/unpersist/checkpoint order may cause serious problem (may cause RDD recomputation, checkpoint take no effect and so on), about this complexity, we can take a look into the VF-BFGS implementation on spark: it use the pattern "persist current step RDDs, then unpersist previous step RDDs" like many other algos in spark mllib. The complexity is at, spark always do lazy computation, when you persist RDD, it do not persist immediately, but postponed to RDD.action called. If the internal code call `unpersist` too early, it will cause the problem that an RDD haven't been computed and haven't been persisted, but already been unpersisted. This feature may be much different than other distributed platform, so that a general interface can really handle this problem correctly and still keep high efficient in the same time? [~sethah] Do you consider this detail problems when you designing the general optimizer interface ? > Vector-free L-BFGS > ------------------ > > Key: SPARK-10078 > URL: https://issues.apache.org/jira/browse/SPARK-10078 > Project: Spark > Issue Type: New Feature > Components: ML > Reporter: Xiangrui Meng > Assignee: Yanbo Liang > > This is to implement a scalable version of vector-free L-BFGS > (http://papers.nips.cc/paper/5333-large-scale-l-bfgs-using-mapreduce.pdf). > Design document: > https://docs.google.com/document/d/1VGKxhg-D-6-vZGUAZ93l3ze2f3LBvTjfHRFVpX68kaw/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org