holdenk commented on a change in pull request #20146: [SPARK-11215][ML] Add
multiple columns support to StringIndexer
URL: https://github.com/apache/spark/pull/20146#discussion_r244378819
##########
File path: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala
##########
@@ -421,3 +603,49 @@ object IndexToString extends
DefaultParamsReadable[IndexToString] {
@Since("1.6.0")
override def load(path: String): IndexToString = super.load(path)
}
+
+/**
+ * A SQL `Aggregator` used by `StringIndexer` to count labels in string
columns during fitting.
+ */
+private class StringIndexerAggregator(numColumns: Int, inputColTypes:
Seq[DataType])
+ extends Aggregator[Row, Array[OpenHashMap[String, Long]],
Array[OpenHashMap[String, Long]]] {
+
+ override def zero: Array[OpenHashMap[String, Long]] =
+ Array.fill(numColumns)(new OpenHashMap[String, Long]())
+
+ def reduce(
+ array: Array[OpenHashMap[String, Long]],
+ row: Row): Array[OpenHashMap[String, Long]] = {
+ for (i <- 0 until numColumns) {
+ val stringValue = row.getString(i)
+ // We don't count for null and NaN values.
Review comment:
Looking at the logic in nullExpression.scala it does seem like the old way
would consider both null and NaN to be null. My only worry is depending on
doing this after string conversion with NaN is if we have different string
reprs for -inf or things like this. Would it make sense to do a fill or is NaN
the guaranteed string outcome?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
