[
https://issues.apache.org/jira/browse/SPARK-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15792615#comment-15792615
]
Steffen Herbold commented on SPARK-18301:
-----------------------------------------
I think if this is a bug or a feature request depends on the point of view.
Since structured types are natively supported by Spark the simple assumption
is, that they are supported by all features of Spark. If they are not supported
by specific features (e.g., transformers), then there should either be a good
reason for this, or it is a bug.
In case there is a reason, this should be part of the documentation and this
should be changed to a feature request. If not, then this constitutes a bug and
if other transformers are also not able to work with them, it might actually be
major instead of minor.
> VectorAssembler does not support StructTypes
> --------------------------------------------
>
> Key: SPARK-18301
> URL: https://issues.apache.org/jira/browse/SPARK-18301
> Project: Spark
> Issue Type: Bug
> Components: MLlib
> Affects Versions: 2.0.1
> Environment: Windows Standalone Mode, Java
> Reporter: Steffen Herbold
> Priority: Minor
>
> I tried to transform a structured type using the VectorAssembler as follows:
> {code:java}
> VectorAssembler va = new VectorAssembler().setInputCols(new String[]
> { "metrics.Line", "metrics.McCC" }).setOutputCol("features");
> dataframe= va.transform(dataframe);
> {code}
> This yields the following exception:
> {code:java}
> Exception in thread "main" java.lang.IllegalArgumentException: Field
> "metrics.McCC" does not exist.
> at
> org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:228)
> at
> org.apache.spark.sql.types.StructType$$anonfun$apply$1.apply(StructType.scala:228)
> at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
> at scala.collection.AbstractMap.getOrElse(Map.scala:59)
> at org.apache.spark.sql.types.StructType.apply(StructType.scala:227)
> at
> org.apache.spark.ml.feature.VectorAssembler$$anonfun$5.apply(VectorAssembler.scala:116)
> at
> org.apache.spark.ml.feature.VectorAssembler$$anonfun$5.apply(VectorAssembler.scala:116)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
> at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
> at
> org.apache.spark.ml.feature.VectorAssembler.transformSchema(VectorAssembler.scala:116)
> at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:70)
> at
> org.apache.spark.ml.feature.VectorAssembler.transform(VectorAssembler.scala:54)
> at
> de.ugoe.cs.smartshark.jobs.DefectPredictionExample.main(DefectPredictionExample.java:53)
> {code}
> The schema of the dataframe is:
> {noformat}
> |-- metrics: struct (nullable = true)
> | |-- Line: double (nullable = true)
> | |-- McCC: double (nullable = true)
> ...
> {noformat}
> The transfomation works, if I first use withColumn to make "metrics.Line" and
> "metrics.McCC" into columns of the dataframe:
> {code:java}
> dataframe.withColumn("Line", data.col("metrics.Line")
> dataframe.withColumn("McCC", data.col("metrics.McCC")
> VectorAssembler va = new VectorAssembler().setInputCols(new String[]
> { "metrics.McCC", "metrics.NL" }).setOutputCol("features");
> fileState = va.transform(dataframe);
> {code}
> However, this workaround is quite costly and direct support to access the
> nested values would be very helpful.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
