Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/20719#discussion_r175321399
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringEvaluator.scala
---
@@ -170,6 +171,13 @@ private[evaluation] abstract class Silhouette {
def overallScore(df: DataFrame, scoreColumn: Column): Double = {
df.select(avg(scoreColumn)).collect()(0).getDouble(0)
}
+
+ protected def getNumberOfFeatures(dataFrame: DataFrame, columnName:
String): Int = {
+ val group =
AttributeGroup.fromStructField(dataFrame.schema(columnName))
+ group.numAttributes.getOrElse {
--- End diff --
We should use `size` of `AttributeGroup` to determine its number of
attributes. When it is -1, we can leverage `first` then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
