Gengliang Wang created SPARK-31935:
--------------------------------------
Summary: Data source options should be propagated in method
`checkAndGlobPathIfNecessary`
Key: SPARK-31935
URL: https://issues.apache.org/jira/browse/SPARK-31935
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.1.0
Reporter: Gengliang Wang
Data source options should be propagated into the hadoop configuration of
method `checkAndGlobPathIfNecessary`
>From org.apache.hadoop.fs.FileSystem.java:
{code:java}
public static FileSystem get(URI uri, Configuration conf) throws IOException {
String scheme = uri.getScheme();
String authority = uri.getAuthority();
if (scheme == null && authority == null) { // use default FS
return get(conf);
}
if (scheme != null && authority == null) { // no authority
URI defaultUri = getDefaultUri(conf);
if (scheme.equals(defaultUri.getScheme()) // if scheme matches default
&& defaultUri.getAuthority() != null) { // & default has authority
return get(defaultUri, conf); // return default
}
}
String disableCacheName = String.format("fs.%s.impl.disable.cache", scheme);
if (conf.getBoolean(disableCacheName, false)) {
return createFileSystem(uri, conf);
}
return CACHE.get(uri, conf);
}
{code}
With this, we can specify URI schema and authority related configurations for
scanning file systems.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
