Sungwon created SPARK-36766:
-------------------------------
Summary: Spark SQL DDL does not recognize fs.s3.impl implied
filesystem in LOCATION tag
Key: SPARK-36766
URL: https://issues.apache.org/jira/browse/SPARK-36766
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.0.2
Reporter: Sungwon
CREATE TABLE with the LOCATION tag via SPARK SQL DDL does not seem to recognize
the fileSystem implied by setting fs.s3.impl.
The environment setting is set to:
|spark.hadoop.fs.s3.impl|org.apache.hadoop.fs.s3a.S3AFileSystem|
The following spark.sql ddl raises an exception:
spark.sql("""CREATE TABLE loca.db.table (
col1 string,
col2 string,
)
USING iceberg
LOCATION 's3://bucket/prefix'
""")
Exception:
Py4JJavaError: An error occurred while calling o86.sql. :
org.apache.iceberg.exceptions.CommitStateUnknownException: Got exception:
org.apache.hadoop.fs.UnsupportedFileSystemException No FileSystem for scheme
"s3"
This is inconsistent with other spark functions such as
spark.read.parquet("s3://bucket/prefix") which succeeds with the same
configuration
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
