[
https://issues.apache.org/jira/browse/HADOOP-18854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825443#comment-17825443
]
Mukund Thakur commented on HADOOP-18854:
----------------------------------------
There is already an option to disable merging
{code:java}
<property>
<name>fs.s3a.vectored.read.max.merged.size</name>
<value>1M</value>
<description>
What is the largest merged read size in bytes such
that we group ranges together during vectored read.
Setting this value to 0 will disable merging of ranges.
</description>
<property>
{code}
> add options to disable range merging of vectored io
> ---------------------------------------------------
>
> Key: HADOOP-18854
> URL: https://issues.apache.org/jira/browse/HADOOP-18854
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs, fs/s3
> Affects Versions: 3.3.5, 3.3.6
> Reporter: Steve Loughran
> Priority: Major
>
> I'm seeing test failures in my PARQUET-2171 pr because assertions about the
> #of bytes read isn't holding -small files are being read and the vector range
> merging is pulling in the whole file.
> ```
> [ERROR] TestInputOutputFormat.testReadWriteWithCounter:338 bytestotal !=
> bytesread expected:<5510> but was:<11020>
> ```
> I think for parquet i will add an option to disable vector io, but really the
> filesystems which support it should allow for merging to be disabled
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
