[
https://issues.apache.org/jira/browse/IGNITE-9180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stuart Macdonald updated IGNITE-9180:
-------------------------------------
Description:
The IgniteSparkSession class extends SparkSession and overrides the
cloneSession() method. The contract for cloneSession() explicitly states that
it should clone all state (ie. the sharedState and sessionState fields),
however the IgniteSparkSession implementation doesn't clone its state fields.
This has the side-effect of breaking stateful operations for anything which
uses cloneSession(), for example a Spark streaming job will not be able to use
cached data across streaming microbatches, which is a significant issue for
such applications.
was:
The IgniteSparkSession class extends SparkState and overrides the
cloneSession() method. The contract for cloneSession() explicitly states that
it should clone all state (ie. the sharedState and sessionState fields),
however the IgniteSparkSession implementation doesn't clone its state fields.
This has the side-effect of breaking stateful operations for anything which
uses cloneSession(), for example a Spark streaming job will not be able to use
cached data across streaming microbatches, which is a significant issue for
such applications.
> IgniteSparkSession Should Copy State on cloneSession()
> ------------------------------------------------------
>
> Key: IGNITE-9180
> URL: https://issues.apache.org/jira/browse/IGNITE-9180
> Project: Ignite
> Issue Type: Bug
> Components: spark
> Reporter: Stuart Macdonald
> Priority: Major
>
> The IgniteSparkSession class extends SparkSession and overrides the
> cloneSession() method. The contract for cloneSession() explicitly states that
> it should clone all state (ie. the sharedState and sessionState fields),
> however the IgniteSparkSession implementation doesn't clone its state fields.
> This has the side-effect of breaking stateful operations for anything which
> uses cloneSession(), for example a Spark streaming job will not be able to
> use cached data across streaming microbatches, which is a significant issue
> for such applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
