Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/12066#discussion_r60953764
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/classification/Classifier.scala ---
@@ -115,7 +116,16 @@ abstract class ClassificationModel[FeaturesType, M <:
ClassificationModel[Featur
}
predictUDF(col(getFeaturesCol))
}
- outputData = outputData.withColumn(getPredictionCol, predUDF)
+ // determine number of classes either from metadata if provided.
+ val labelSchema = dataset.schema($(labelCol))
+ // extract label metadata from label column if present, or create a
nominal attribute
+ // to output the number of labels
+ val labelAttribute = Attribute.fromStructField(labelSchema) match {
--- End diff --
1. Thinking more about this, we should definitely prepare the
labelAttribute during model fitting/construction. During fitting, we know the
schema of the output column (since it matches the label column, except for the
name). We should not use the labelCol schema during transformation since that
could be a completely different label (thought that's unlikely).
2. The attribute name is supposed to match the column name. The reason we
need this duplicated info is for AttributeGroups, which store attributes for
each element in a Vector column.
3. By "compute it in validateAndTransformSchema," I mean that we should
make the output schema during schema validation as precise as possible. In
this case, that means we should add metadata to the output column in
transformSchema. I suspect you can create a helper method in ClassifierParams
which computes the metadata so that you can use the same code path for both
schema validation and the actual transform() method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
