zhengruifeng created SPARK-25989:
------------------------------------
Summary: OneVsRestModel handle empty outputCols incorrectly
Key: SPARK-25989
URL: https://issues.apache.org/jira/browse/SPARK-25989
Project: Spark
Issue Type: Bug
Components: ML
Affects Versions: 3.0.0
Reporter: zhengruifeng
{\{ml.classification.ClassificationModel}} will ignore empty output columns.
However, \{{OneVsRestModel}} still try to append new column even if its name is
an empty string.
{code:java}
scala> ovrModel.setPredictionCol("").transform(test).show
+-----+--------------------+--------------------+---+
|label| features| rawPrediction| |
+-----+--------------------+--------------------+---+
| 0.0|(4,[0,1,2,3],[-0....|[-0.0965652626152...|2.0|
| 0.0|(4,[0,1,2,3],[-0....|[0.07880609384635...|2.0|
| 0.0|(4,[0,1,2,3],[-1....|[0.01891571586984...|2.0|
| 0.0|(4,[0,1,2,3],[0.1...|[0.72409973016524...|0.0|
| 0.0|(4,[0,1,2,3],[0.1...|[0.48045978946729...|2.0|
| 0.0|(4,[0,1,2,3],[0.3...|[1.05496616040758...|0.0|
| 0.0|(4,[0,1,2,3],[0.3...|[0.79508659065535...|0.0|
| 0.0|(4,[0,1,2,3],[0.6...|[1.47437469552081...|0.0|
| 0.0|(4,[0,1,2,3],[0.6...|[1.23302929670223...|0.0|
| 0.0|(4,[0,1,2,3],[0.8...|[1.79816156359706...|0.0|
| 1.0|(4,[0,1,2,3],[-0....|[-3.1564309664080...|1.0|
| 1.0|(4,[0,1,2,3],[-0....|[-3.2217906250571...|1.0|
| 1.0|(4,[0,1,2,3],[-0....|[-2.9171126308553...|1.0|
| 1.0|(4,[0,1,2,3],[-0....|[-2.8316993051998...|1.0|
| 2.0|(4,[0,1,2,3],[-0....|[-1.6486206847760...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.9252139721697...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.9025379528484...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.8518243169707...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-1.0990190524225...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.9973479746889...|2.0|
+-----+--------------------+--------------------+---+
only showing top 20 rows
scala>
ovrModel.setPredictionCol("").setRawPredictionCol("raw").transform(test).show
+-----+--------------------+--------------------+---+
|label| features| raw| |
+-----+--------------------+--------------------+---+
| 0.0|(4,[0,1,2,3],[-0....|[-0.0965652626152...|2.0|
| 0.0|(4,[0,1,2,3],[-0....|[0.07880609384635...|2.0|
| 0.0|(4,[0,1,2,3],[-1....|[0.01891571586984...|2.0|
| 0.0|(4,[0,1,2,3],[0.1...|[0.72409973016524...|0.0|
| 0.0|(4,[0,1,2,3],[0.1...|[0.48045978946729...|2.0|
| 0.0|(4,[0,1,2,3],[0.3...|[1.05496616040758...|0.0|
| 0.0|(4,[0,1,2,3],[0.3...|[0.79508659065535...|0.0|
| 0.0|(4,[0,1,2,3],[0.6...|[1.47437469552081...|0.0|
| 0.0|(4,[0,1,2,3],[0.6...|[1.23302929670223...|0.0|
| 0.0|(4,[0,1,2,3],[0.8...|[1.79816156359706...|0.0|
| 1.0|(4,[0,1,2,3],[-0....|[-3.1564309664080...|1.0|
| 1.0|(4,[0,1,2,3],[-0....|[-3.2217906250571...|1.0|
| 1.0|(4,[0,1,2,3],[-0....|[-2.9171126308553...|1.0|
| 1.0|(4,[0,1,2,3],[-0....|[-2.8316993051998...|1.0|
| 2.0|(4,[0,1,2,3],[-0....|[-1.6486206847760...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.9252139721697...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.9025379528484...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.8518243169707...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-1.0990190524225...|2.0|
| 2.0|(4,[0,1,2,3],[-0....|[-0.9973479746889...|2.0|
+-----+--------------------+--------------------+---+
only showing top 20 rows
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
