[
https://issues.apache.org/jira/browse/SPARK-60807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ruifeng Zheng updated SPARK-60807:
----------------------------------
Description:
Fetching a Spark Connect ML training-summary attribute can fail with
CONNECT_ML.MODEL_SUMMARY_LOST after model offloading.
The Python client recovers by sending CreateSummary and then retrying FETCH
once. The in-memory ML cache uses soft values, so an entry can be evicted again
between those RPCs. The retry then reloads a model without its summary.
Relation FETCH has a related gap: it restores the summary and then performs
another cache lookup through AttributeHelper.
A deterministic reproduction is to evict a fitted model before each command
FETCH, or configure the server ML cache with a zero in-memory size limit.
Summary access should succeed while the model remains available on disk.
The server should restore the summary and evaluate the attribute on the same
strongly referenced model instance. Saving the dataset plan from the existing
CreateSummary request beside the offloaded model allows later FETCH requests to
restore summaries after further evictions, without changing the protocol.
was:
Fetching a Spark Connect ML training-summary attribute can fail with
CONNECT_ML.MODEL_SUMMARY_LOST after model offloading.
The Python client recovers by sending CreateSummary and then retrying FETCH
once. The in-memory ML cache uses soft values, so an entry can be evicted again
between those RPCs. The retry then reloads a model without its summary.
Relation FETCH has a related gap: it restores the summary and then performs
another cache lookup through AttributeHelper.
A deterministic reproduction is to evict a fitted model before each command
FETCH, or configure the server ML cache with a zero in-memory size limit.
Summary access should succeed while the model remains available on disk.
Restore the summary and evaluate the attribute on the same strongly referenced
model instance within one FETCH request. The command should carry the summary
dataset, as relation FETCH already does, while retaining compatibility with
older clients and servers.
> Restore offloaded ML model summaries during fetch
> -------------------------------------------------
>
> Key: SPARK-60807
> URL: https://issues.apache.org/jira/browse/SPARK-60807
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Ruifeng Zheng
> Priority: Major
> Labels: pull-request-available
>
> Fetching a Spark Connect ML training-summary attribute can fail with
> CONNECT_ML.MODEL_SUMMARY_LOST after model offloading.
> The Python client recovers by sending CreateSummary and then retrying FETCH
> once. The in-memory ML cache uses soft values, so an entry can be evicted
> again between those RPCs. The retry then reloads a model without its summary.
> Relation FETCH has a related gap: it restores the summary and then performs
> another cache lookup through AttributeHelper.
> A deterministic reproduction is to evict a fitted model before each command
> FETCH, or configure the server ML cache with a zero in-memory size limit.
> Summary access should succeed while the model remains available on disk.
> The server should restore the summary and evaluate the attribute on the same
> strongly referenced model instance. Saving the dataset plan from the existing
> CreateSummary request beside the offloaded model allows later FETCH requests
> to restore summaries after further evictions, without changing the protocol.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
