Hi,
I am trying to run random forest classification by using Spark ML api but I
am having issues with creating right data frame input into pipeline.
Here is sample data:
age,hours_per_week,education,sex,salaryRange
38,40,"hs-grad","male","A"
28,40,"bachelors","female","A"
52,45,"hs-grad","male","B"
31,50,"masters","female","B"
42,40,"bachelors","male","B"
age and hours_per_week are integers while other features including label
salaryRange are categorical (String)
Loading this csv file (lets call it sample.csv) can be done by Spark csv
library like this:
val data = sqlContext.csvFile("/home/dusan/sample.csv")
By default all columns are imported as string so we need to change "age" and
"hours_per_week" to Int:
val toInt = udf[Int, String]( _.toInt)
val dataFixed = data.withColumn("age",
toInt(data("age"))).withColumn("hours_per_week",toInt(data("hours_per_week")))
Just to check how schema looks now:
scala> dataFixed.printSchema
root
|-- age: integer (nullable = true)
|-- hours_per_week: integer (nullable = true)
|-- education: string (nullable = true)
|-- sex: string (nullable = true)
|-- salaryRange: string (nullable = true)
Then lets set the cross validator and pipeline:
val rf = new RandomForestClassifier()
val pipeline = new Pipeline().setStages(Array(rf))
val cv = new
CrossValidator().setNumFolds(10).setEstimator(pipeline).setEvaluator(new
BinaryClassificationEvaluator)
Error shows up when running this line:
val cmModel = cv.fit(dataFixed)
*java.lang.IllegalArgumentException: Field "features" does not exist.*
It is possible to set label column and feature column in
RandomForestClassifier ,however I have 4 columns as predictors (features)
not only one.
How I should organize my data frame so it has label and features columns
organized correctly?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-create-correct-data-frame-for-classification-in-Spark-ML-tp23490.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
