Gengliang Wang created SPARK-61910:
--------------------------------------
Summary: Let native data sources access the file systems that
Spark is configured with
Key: SPARK-61910
URL: https://issues.apache.org/jira/browse/SPARK-61910
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Gengliang Wang
A native library gets nothing from Spark to access storage: {{SparkDsHost}}
only has the versions of Spark and of the interface, and the logger, and a data
source only gets the options of the user in {{SparkDsCreateInfo}}. So a native
source that reads files, such as the Parquet source of the Rust SDK examples
(https://github.com/apache/spark/pull/59692), reads local files only: it has
neither the Hadoop file systems, nor their configuration ({{spark.hadoop.*}}),
nor the credentials that the readers of Spark use, unless the user passes them
as options.
Proposal: decide how a library accesses the storage that Spark is configured
with, for example by passing the Hadoop configuration of the session to the
library, or with host functions that open and read files through the file
systems of Spark, and expose it in the Rust SDK (SPARK-61282). Both can be
added within version 1 of the interface ({{spark_datasource.h}}), as fields
appended to its structs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]