[
https://issues.apache.org/jira/browse/ARROW-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Dimitrov updated ARROW-5318:
---------------------------------
Description:
When reading using HdfsFilesystem's read method, the amount is not constant and
flactuates between extra 0% to 300%.
Example code
{code:java}
fs = hdfs.connect(hostname, driver='libhdfs')
f = fs.open(dataset_path)
f.read(nbytes=3500000){code}
In this case, the read can send back up to 15 M bytes. The issue is true with
'libhdfs3' as well. Also present in newer versions of pyarrow.
was:
When reading using HdfsFilesystem's read method, the amount is not constant and
flactuates between extra 0% to 300%.
Example code
{code:java}
fs = hdfs.connect(hostname, driver='libhdfs')
f = fs.open(dataset_path)
f.read(nbytes=3500000){code}
In this case, the read can send back up to 15 M bytes. The issue is true with
'libhdfs3' as well. Also present in newer versions of pyarrow.
> pyarrow hdfs reader overrequests
> ----------------------------------
>
> Key: ARROW-5318
> URL: https://issues.apache.org/jira/browse/ARROW-5318
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.10.0
> Reporter: Ivan Dimitrov
> Priority: Blocker
>
> When reading using HdfsFilesystem's read method, the amount is not constant
> and flactuates between extra 0% to 300%.
> Example code
> {code:java}
> fs = hdfs.connect(hostname, driver='libhdfs')
> f = fs.open(dataset_path)
> f.read(nbytes=3500000){code}
> In this case, the read can send back up to 15 M bytes. The issue is true with
> 'libhdfs3' as well. Also present in newer versions of pyarrow.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
