[
https://issues.apache.org/jira/browse/BEAM-9152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540343#comment-17540343
]
Mohammad Saleh Refahi commented on BEAM-9152:
---------------------------------------------
Hi Jean, I had tried it , but I faced with new error when I changed the
length=end - start + 1 --> length=end - start.
So, what I understand from above error , there is a limitation in buffer size ,
which I think it is related to io.buffer.driver. I found it 1 months ago , so I
can't say the exact definition how I understood it.
For me , I understood the size of buffer is limited to 20mb (2097152b). To deal
with this problem, I had to break my data up to multiple shards which the size
of each shards is less than 20mb.
I am not sure about my solution , so I may be faced with a new error in the
future.
> Hadoop Downloader Range not correct
> -----------------------------------
>
> Key: BEAM-9152
> URL: https://issues.apache.org/jira/browse/BEAM-9152
> Project: Beam
> Issue Type: Bug
> Components: io-py-tfrecord
> Reporter: Fangyuan Zhou
> Priority: P3
> Fix For: Missing
>
>
> I found that the `HdfsDownloader.get_range(self, start, end)' get range
> {color:#ff0000}[start, end]{color} rather than {color:#ff0000}[start,
> end){color}. This will cause an error while reading hdfs file.
>
> File
> "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/tfrecordio.py",
> line 127, in read_record
> buf = file_handle.read(buf_length_expected)
> File
> "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/filesystemio.py",
> line 123, in readinto
> b[:len(data)] = data
> ValueError: memoryview assignment: lvalue and rvalue have different
> structures
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
