[
https://issues.apache.org/jira/browse/HADOOP-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642272#comment-17642272
]
ASF GitHub Bot commented on HADOOP-18543:
-----------------------------------------
masteryhx commented on PR #5172:
URL: https://github.com/apache/hadoop/pull/5172#issuecomment-1334751421
> sorry, but I'm going to say -1 to using the normal IO buffer size as the
GET range. The default value of 4k is way too small even for parquet/orc reads,
it will break all existing apps in performance terms: distcp, parquet library,
avro, ORC, everything, as they all use the default value.
>
> 1. there is a configuration option for multipart download size, which is
filesystem-wide. Not as flexible, but something everyone will expect to work.
> 2. If you want better control of read policy, buffer sizes etc, then this
connector needs to implement openFile(), as s3a and abfs do. that will let you
add a new option to specify the range for GET calls.
Thanks a lot for your detailed explaination and suggestion.
I agree that it's better not to change the implementaion of current
interface.
I'd like to make oss also implement openFile() in this pr as s3a does which
could also meet our needs.
WDYT?
> AliyunOSS: AliyunOSSFileSystem#open(Path path, int bufferSize) should use
> buffer size as its downloadPartSize
> -------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-18543
> URL: https://issues.apache.org/jira/browse/HADOOP-18543
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/oss
> Reporter: Hangxiang Yu
> Priority: Major
> Labels: pull-request-available
>
> In our application, different components have their own suitable buffer size
> to download.
> But currently, AliyunOSSFileSystem#open(Path path, int bufferSize) just get
> downloadPartSize from configuration.
> We cannnot use different value for different components in our programs.
> I think we should the method should use the buffer size from the paramater.
> AliyunOSSFileSystem#open(Path path) could have default value as current
> default downloadPartSize.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
