[
https://issues.apache.org/jira/browse/IMPALA-10695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17347435#comment-17347435
]
ASF subversion and git services commented on IMPALA-10695:
----------------------------------------------------------
Commit 26eaa43dbe6f99a6f97f4d3e0eafb1cf7c24217c in impala's branch
refs/heads/master from Yong Yang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=26eaa43 ]
IMPALA-10695: add dedicated thread pool for OSS/JindoFS.
OSS is the object store in Alibaba cloud, just like s3a,
and jindofs is a gateway based on Alibaba cloud object store.
The following is about the JindoFS, for more information:
https://www.alibabacloud.com/blog/introducing-jindofs-a-high-performance-data-lake-storage-solution_595600
If Alibaba object store would be treated as local disk
without this change, the query performance is not good.
This change would create a dedicated queue for this kind of target,
and improved the OSS scan performance.
I have tested it in our environment,
and observed at least double the scan speed.
New flags:
- num_oss_io_threads: Number of OSS/JindoFS I/O threads. Defaults to 16.
Change-Id: I4643105628f3860e3145c85d9ed205fe20291add
Signed-off-by: Yong Yang <[email protected]>
Reviewed-on: http://gerrit.cloudera.org:8080/17455
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> impala need use an independent disk io queue for JindoFS
> --------------------------------------------------------
>
> Key: IMPALA-10695
> URL: https://issues.apache.org/jira/browse/IMPALA-10695
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Yong Yang
> Assignee: Yong Yang
> Priority: Critical
>
> So far, the S3A, Adls and ABFS have dedicate queues in impala IO path. Ali
> cloud has similar target, like oss and jfs.
> To improve the query IO performance when target files are stored in oss/jfs,
> it is necessary to create a dedicate queue for that.
>
> BTW, I have some code ready for that when I am optimizing this situation.
> Please share your opinion if I should commit that part code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
