[
https://issues.apache.org/jira/browse/ARROW-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17659031#comment-17659031
]
Rok Mihevc commented on ARROW-2002:
-----------------------------------
This issue has been migrated to [issue
#17984|https://github.com/apache/arrow/issues/17984] on GitHub. Please see the
[migration documentation|https://github.com/apache/arrow/issues/14542] for
further details.
> use pyarrow download file will raise queue.Full exceptions sometimes
> --------------------------------------------------------------------
>
> Key: ARROW-2002
> URL: https://issues.apache.org/jira/browse/ARROW-2002
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.8.0
> Environment: operating system: all
> platform: all
> Reporter: kmiku7
> Assignee: kmiku7
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> When we download file from hdfs, if the speed writer thread write data is
> slower than read speed, download() will raise queue.Fulll exceptions, because
> write_queue is full.
> I think when we download file, we can wait until write_queue has space to
> enqueue new item if writer_thread is alive. Like what upload() does.
> {code}
> >>> import pyarrow as pa
> >>> cli = pa.hdfs.connect(user='USERNAME')
> >>> cli.download('/REMOTE/HDFS/PATH', '/LOCAL/FILE/PATH')
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "pyarrow/io-hdfs.pxi", line 428, in
> pyarrow.lib.HadoopFileSystem.download
> (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66399)
> File "pyarrow/io-hdfs.pxi", line 429, in
> pyarrow.lib.HadoopFileSystem.download
> (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:66351)
> File "pyarrow/io.pxi", line 315, in pyarrow.lib.NativeFile.download
> (/arrow/python/build/temp.linux-x86_64-3.4/lib.cxx:52249)
> File "/usr/lib/python3.4/queue.py", line 187, in put_nowait
> return self.put(item, block=False)
> File "/usr/lib/python3.4/queue.py", line 133, in put
> raise Full
> queue.Full
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
