[
https://issues.apache.org/jira/browse/IMPALA-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695842#comment-17695842
]
ASF subversion and git services commented on IMPALA-11920:
----------------------------------------------------------
Commit 99d676f8fb71304838c8fde70d3dd220f8f1f52a in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=99d676f8f ]
IMPALA-11920: Support spill to HDFS address by service ID
Allows addressing HDFS (and Ozone) filesystems in `scratch_dirs` by a
service identifier that doesn't include a port number. Examples
- "hdfs://hdfs1/:10G" uses the root directory of HDFS with a 10G limit
- "ofs://ozone1/tmp::" uses /tmp in Ozone with default limit/priority
Updates `scratch_dirs` parsing to allow whitespace after each specifier,
as in "hdfs://hdfs1/ , /tmp". This is unambiguous and avoids failures
for simple mistakes.
Testing:
- new backend test cases run with HDFS and Ozone
- manually tested that Impala starts with
--impalad_args=--scratch_dirs=ofs://localhost/tmp,/tmp
creates impala-scratch in both locations
Change-Id: Ie069cba211df85fe90d174900b20a26fcc1f7167
Reviewed-on: http://gerrit.cloudera.org:8080/19496
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Michael Smith <[email protected]>
> Spill to HDFS/Ozone can't address by service name
> -------------------------------------------------
>
> Key: IMPALA-11920
> URL: https://issues.apache.org/jira/browse/IMPALA-11920
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.2.0
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
>
> HDFS and Ozone HA deployments often use a service name, such as
> {{ofs://ozone1/...}}. The current {{scratch_dirs}} token parsing for requires
> a {{hostname:port}}, which doesn't support HA addressing. We should make the
> parsing smarter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
