[
https://issues.apache.org/jira/browse/SPARK-30851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038917#comment-17038917
]
Vladislav Glinskiy commented on SPARK-30851:
--------------------------------------------
This issue can be bypassed by caching loaded instances paths from
{{LoadInstanceStart}} events using {{reader}} instance as a key.
{code:java}
private[this] val loadedInstancePaths = new mutable.WeakHashMap[MLReader[_],
String]()
...
case loadInstanceStart: LoadInstanceStart[_] =>
loadedInstancePaths.put(loadInstanceStart.reader, loadInstanceStart.path)
case loadInstanceEnd: LoadInstanceEnd[_] => val path =
loadedInstancePaths.get(loadInstanceEnd.reader)
...
{code}
> Add 'path' field to the 'LoadInstanceEnd' ML listener event
> -----------------------------------------------------------
>
> Key: SPARK-30851
> URL: https://issues.apache.org/jira/browse/SPARK-30851
> Project: Spark
> Issue Type: Improvement
> Components: ML
> Affects Versions: 3.0.0
> Reporter: Vladislav Glinskiy
> Priority: Major
> Fix For: 3.0.0
>
>
> The
> [LoadInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L92]
> ML listener event that is added on the scope of SPARK-23674 has no 'path'
> field which makes it impossible to determine from what path an ML instance
> was loaded as well as there is no way to get instance's 'uid' via
> [LoadInstanceStart|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L84]
> event.
>
> The
> [LoadInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L92]
> must be changed to include 'path' field. Please, refer
> [SaveInstanceStart|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L101]
> and
> [SaveInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L109]
> events. Both of them have `path` but
> [LoadInstanceStart|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L84]
> and
> [LoadInstanceEnd|https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/events.scala#L92]
> are not.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
