Yong Yang has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17455 )
Change subject: IMPALA-10695: add dedicated thread pool for OSS/JindoFS. ...................................................................... IMPALA-10695: add dedicated thread pool for OSS/JindoFS. OSS is the object store in Alibaba cloud, just like s3a, and jindofs is a gateway based on Alibaba cloud object store. The following is about the JindoFS, for more information: https://www.alibabacloud.com/blog/introducing-jindofs-a-high-performance-data-lake-storage-solution_595600 If Alibaba object store would be treated as local disk without this change, the query performance is not good. This change would create a dedicated queue for this kind of target, and improved the OSS scan performance. I have tested it in our environment, and observed at least double the scan speed. New flags: - num_oss_io_threads: Number of OSS/JindoFS I/O threads. Defaults to 16. Change-Id: I4643105628f3860e3145c85d9ed205fe20291add Signed-off-by: Yong Yang <yong.su...@163.com> --- M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h 5 files changed, 25 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/17455/6 -- To view, visit http://gerrit.cloudera.org:8080/17455 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4643105628f3860e3145c85d9ed205fe20291add Gerrit-Change-Number: 17455 Gerrit-PatchSet: 6 Gerrit-Owner: Yong Yang <yong.su...@163.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Yong Yang <yong.su...@163.com>
