Yida Wu created IMPALA-10945:
--------------------------------
Summary: Unfavorable change on the S3 scratch path behavior
Key: IMPALA-10945
URL: https://issues.apache.org/jira/browse/IMPALA-10945
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 4.1.0
Reporter: Yida Wu
Assignee: Yida Wu
IMPALA-10429 "Support Spill to HDFS" introduces a new behavior to S3 scratch
path.
It added a connection verification logic for the S3 path (for optimization),
but the HdfsFsCache requires the path to have a directory behind, like
"s3a://host/dir", otherwise it will throw an error.
[https://github.com/apache/impala/blob/1a61a8025c87c37921a1bba4c49f754d8bd10bcc/be/src/runtime/tmp-file-mgr.cc#L333]
The change brings a problem, that may require the user to have a different s3a
scratch path. It would be good to be compatible with the path we support in the
previous version.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)