[
https://issues.apache.org/jira/browse/SPARK-45600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allison Wang updated SPARK-45600:
---------------------------------
Description: Currently, registered data sources are stored in `sharedState`
and can be accessed across multiple sessions. This, however, will not work with
Spark Connect. We should make this registration session level, and support
static registration (e.g. using pip install) in the future. (was: Currently we
have added a few instance variables to store information for Python data source
reader. We should have a dedicated reader class for Python data source to make
the current DataFrameReader clean.)
> Make Python data source registration session level
> --------------------------------------------------
>
> Key: SPARK-45600
> URL: https://issues.apache.org/jira/browse/SPARK-45600
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
> Affects Versions: 4.0.0
> Reporter: Allison Wang
> Priority: Major
>
> Currently, registered data sources are stored in `sharedState` and can be
> accessed across multiple sessions. This, however, will not work with Spark
> Connect. We should make this registration session level, and support static
> registration (e.g. using pip install) in the future.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
