[
https://issues.apache.org/jira/browse/IMPALA-11448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated IMPALA-11448:
-------------------------------------
Attachment: Screen Shot 2022-07-20 at 10.35.24 AM.png
> Always assign Ozone I/O to remote thread group to improve performance
> ---------------------------------------------------------------------
>
> Key: IMPALA-11448
> URL: https://issues.apache.org/jira/browse/IMPALA-11448
> Project: IMPALA
> Issue Type: Bug
> Affects Versions: Impala 4.0.0
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Attachments: Screen Shot 2022-07-20 at 10.35.24 AM.png
>
>
> IMPALA-9400 added initial support for Ozone (o3fs/ofs) by assuming all Ozone
> I/O as remote, which is a valid assumption.
> However, the Impala's internal logic will assign the I/O to a single local
> disk I/O thread , severely limiting the I/O parallelism. This is evident when
> running the debug build, which fails at the following check:
> {noformat}
> Log file created at: 2022/07/18 18:15:02
> Running on machine: rhel05.ozone.cisco.local
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> F0718 18:15:02.269232 105827 disk-io-mgr.cc:561]
> 004a5f3dd4a34435:f2e0cb9b00000030] Check failed: !IsOzonePath(file)
> F0718 18:15:02.269235 105832 disk-io-mgr.cc:561]
> 004a5f3dd4a34435:f2e0cb9b00000003] Check failed: !IsOzonePath(file) F0718
> 18:15:02.269273 105834 disk-io-mgr.cc:561] 004a5f3dd4a34435:f2e0cb9b00000014]
> Check failed: !IsOzonePath(file)
> F0718 18:15:02.269235 105832 disk-io-mgr.cc:561]
> 004a5f3dd4a34435:f2e0cb9b00000003] Check failed: !IsOzonePath(file) F0718
> 18:15:02.269273 105834 disk-io-mgr.cc:561] 004a5f3dd4a34435:f2e0cb9b00000014]
> Check failed: !IsOzonePath(file)
> {noformat}
> The is_remote parameter of a scan range is always false for Ozone:
> {noformat}
> TScanRangeParams {
> 01: scan_range (struct) = TScanRange {
> 01: hdfs_file_split (struct) = THdfsFileSplit {
> 01: relative_path (string) =
> "base_1/2b4595d335caddf5-4c9efd320000000e_1114488364_data.0.parq",
> 02: offset (i64) = 0,
> 03: length (i64) = 15982993,
> 04: partition_id (i64) = 172,
> 05: file_length (i64) = 15982993,
> 06: file_compression (i32) = 0,
> 07: mtime (i64) = 1657968669139,
> 08: is_erasure_coded (bool) = false,
> 09: partition_path_hash (i32) = -343315716,
> },
> },
> 02: volume_id (i32) = 65535,
> 03: try_hdfs_cache (bool) = false,
> 04: is_remote (bool) = false,
> {noformat}
> Because Ozone does not yet have short circuit read support, I think a quick
> fix is to always force Ozone to use the remote I/O thread group assigned io
> it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
