[
https://issues.apache.org/jira/browse/ARROW-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16840647#comment-16840647
]
Ivan Dimitrov commented on ARROW-5318:
--------------------------------------
@[pitrou|https://github.com/apache/arrow/commits?author=pitrou] any thoughts
> pyarrow hdfs reader overrequests
> ----------------------------------
>
> Key: ARROW-5318
> URL: https://issues.apache.org/jira/browse/ARROW-5318
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.10.0
> Reporter: Ivan Dimitrov
> Priority: Blocker
>
> When reading using HdfsFilesystem's read method, the amount is not constant
> and flactuates between extra 0% to 300%.
> Example code
> {code:java}
> fs = hdfs.connect(hostname, driver='libhdfs')
> f = fs.open(dataset_path)
> f.read(nbytes=3500000){code}
> In this case, the read can send back up to 15 M bytes. The issue is true with
> 'libhdfs3' as well. Also present in newer versions of pyarrow.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
