Joseph K. Bradley created SPARK-3903:
----------------------------------------
Summary: Create general data loading method for LabeledPoints
Key: SPARK-3903
URL: https://issues.apache.org/jira/browse/SPARK-3903
Project: Spark
Issue Type: Improvement
Components: MLlib
Reporter: Joseph K. Bradley
Priority: Minor
Proposal: Provide a more general data loading function for LabeledPoints.
* load multiple data files (e.g., train + test), and ensure they have the same
number of features (determined based on a scan of the data)
* use same function for multiple input formats
Proposed function format (in MLUtils), with default parameters:
{code}
def loadLabeledPointsFiles(
sc: SparkContext,
paths: Seq[String],
numFeatures = -1,
vectorFormat = "auto",
numPartitions = sc.defaultMinPartitions): Seq[RDD[LabeledPoint]]
{code}
About the parameters:
* paths: list of paths to data files or folders with data files
* vectorFormat options: dense/sparse/auto
* numFeatures, numPartitions: same behavior as loadLibSVMFile
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
