Gengliang Wang created SPARK-61910:
--------------------------------------

             Summary: Let native data sources access the file systems that 
Spark is configured with
                 Key: SPARK-61910
                 URL: https://issues.apache.org/jira/browse/SPARK-61910
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


A native library gets nothing from Spark to access storage: {{SparkDsHost}} 
only has the versions of Spark and of the interface, and the logger, and a data 
source only gets the options of the user in {{SparkDsCreateInfo}}. So a native 
source that reads files, such as the Parquet source of the Rust SDK examples 
(https://github.com/apache/spark/pull/59692), reads local files only: it has 
neither the Hadoop file systems, nor their configuration ({{spark.hadoop.*}}), 
nor the credentials that the readers of Spark use, unless the user passes them 
as options.

Proposal: decide how a library accesses the storage that Spark is configured 
with, for example by passing the Hadoop configuration of the session to the 
library, or with host functions that open and read files through the file 
systems of Spark, and expose it in the Rust SDK (SPARK-61282). Both can be 
added within version 1 of the interface ({{spark_datasource.h}}), as fields 
appended to its structs.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to