[
https://issues.apache.org/jira/browse/SPARK-16348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359634#comment-15359634
]
Joseph K. Bradley commented on SPARK-16348:
-------------------------------------------
(Thanks [~timhunter] for figuring this out!)
> pyspark.ml MLSerDe should be called using full classpath
> --------------------------------------------------------
>
> Key: SPARK-16348
> URL: https://issues.apache.org/jira/browse/SPARK-16348
> Project: Spark
> Issue Type: Bug
> Components: ML, PySpark
> Affects Versions: 2.0.0
> Reporter: Joseph K. Bradley
> Priority: Critical
>
> Depending on how Spark is set up, pyspark.ml may or may not be able to find
> the MLSerDe instance when referenced as {{sc._jvm.MLSerDe}}. This can cause
> failures {{'JavaPackage' object is not callable}} when trying to access
> Vector or Matrix values from pyspark, such as retrieving the coefficients of
> a LinearRegressionModel.
> Proposal: Whenever we reference a class in the _jvm from pyspark, we should
> use the full classpath: {{sc._jvm.org.apache.spark.ml.python.MLSerDe}}. This
> fixes the bug in my case.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
