Junpei Zhou created SPARK-31400:
-----------------------------------
Summary: The catalogString doesn't distinguish Vectors in ml and
mllib
Key: SPARK-31400
URL: https://issues.apache.org/jira/browse/SPARK-31400
Project: Spark
Issue Type: Bug
Components: ML, MLlib
Affects Versions: 2.4.5
Environment: Ubuntu 16.04
Reporter: Junpei Zhou
h2. Bug Description
The `catalogString` is not detailed enough to distinguish the
pyspark.ml.linalg.Vectors and pyspark.mllib.linalg.Vectors.
h2. How to reproduce the bug
[Here|https://spark.apache.org/docs/latest/ml-features#minmaxscaler] is an
example from the official document (Python code). If I keep all other lines
untouched, and only modify the Vectors import line, which means:
{code:java}
# from pyspark.ml.linalg import Vectors
from pyspark.mllib.linalg import Vectors
{code}
Or you can directly execute the following code snippet:
{code:java}
from pyspark.ml.feature import MinMaxScaler
# from pyspark.ml.linalg import Vectors
from pyspark.mllib.linalg import Vectors
dataFrame = spark.createDataFrame([
(0, Vectors.dense([1.0, 0.1, -1.0]),),
(1, Vectors.dense([2.0, 1.1, 1.0]),),
(2, Vectors.dense([3.0, 10.1, 3.0]),)
], ["id", "features"])
scaler = MinMaxScaler(inputCol="features", outputCol="scaledFeatures")
scalerModel = scaler.fit(dataFrame)
{code}
It will raise an error:
{code:java}
IllegalArgumentException: 'requirement failed: Column features must be of type
struct<type:tinyint,size:int,indices:array<int>,values:array<double>> but was
actually struct<type:tinyint,size:int,indices:array<int>,values:array<double>>.'
{code}
However, the actually struct and the desired struct are exactly the same
string, which cannot provide useful information to the programmer. I would
suggest making the catalogString distinguish pyspark.ml.linalg.Vectors and
pyspark.mllib.linalg.Vectors.
Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
