[
https://issues.apache.org/jira/browse/SPARK-60807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng resolved SPARK-60807.
-----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 60022
[https://github.com/apache/spark/pull/60022]
> Restore offloaded ML model summaries during fetch
> -------------------------------------------------
>
> Key: SPARK-60807
> URL: https://issues.apache.org/jira/browse/SPARK-60807
> Project: Spark
> Issue Type: Bug
> Components: Connect, ML
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Ruifeng Zheng
> Assignee: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Fetching a Spark Connect ML training-summary attribute can fail with
> CONNECT_ML.MODEL_SUMMARY_LOST after model offloading.
> The Python client recovers by sending CreateSummary and then retrying FETCH
> once. The in-memory ML cache uses soft values, so an entry can be evicted
> again between those RPCs. The retry then reloads a model without its summary.
> Relation FETCH has a related gap: it restores the summary and then performs
> another cache lookup through AttributeHelper.
> A deterministic reproduction is to evict a fitted model before each command
> FETCH, or configure the server ML cache with a zero in-memory size limit.
> Summary access should succeed while the model remains available on disk.
> The server should restore the summary and evaluate the attribute on the same
> strongly referenced model instance. Saving the dataset plan from the existing
> CreateSummary request beside the offloaded model allows later FETCH requests
> to restore summaries after further evictions, without changing the protocol.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
