It seems to me that it is the latter. A quick example is the predict method of SparkLogisticRegression ( https://github.com/lensacom/sparkit-learn/blob/master/splearn/linear_model/logistic.py#L139 )
The input is an ArrayRDD which is a wrapper around the RDD in spark, but with numpy-like operations. So while doing a map across this ArrayRDD, each element is passed to the predict method of sklearn's LogReg and the predicted labels are returned as another ArrayRDD. On Sat, Sep 12, 2015 at 9:13 PM, Sebastian Raschka <se.rasc...@gmail.com> wrote: > Interesting! Is it (sparkit-learn) a Python wrapper for the Spark Scala > code (e.g., like PySpark & Mlib) or is it running scikit-learn Python code > on distributed systems? > > On Sep 12, 2015, at 8:54 PM, Andreas Mueller <t3k...@gmail.com> wrote: > > > > On 09/12/2015 04:56 PM, Rex X wrote: > > This project looks interesting > https://github.com/lensacom/sparkit-learn > <https://github.com/lensacom/sparkit-learn/blob/master/README.rst> > > and a nice coded project name :) > > In sparkit-learn, the learning either happens on a single machine, > or separate models are trained on several machines and then ensembled, > right? > > ------------------------------------------------------------------------------ > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Scikit-learn-general mailing list > Scikit-learn-general@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/scikit-learn-general > > -- Godspeed, Manoj Kumar, http://manojbits.wordpress.com <http://goog_1017110195> http://github.com/MechCoder
------------------------------------------------------------------------------
_______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
