[
https://issues.apache.org/jira/browse/BEAM-6380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16736360#comment-16736360
]
Ahmet Altay commented on BEAM-6380:
-----------------------------------
Error is:
*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/gcp/gcsio.py",
line 581, in _start_upload*22:01:22*
self._client.objects.Insert(self._insert_request,
upload=self._upload)*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py",
line 1154, in Insert*22:01:22* upload=upload,
upload_config=upload_config)*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/build/gradleenv/1327086738/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
line 715, in _RunMethod*22:01:22* http_request,
client=self.client)*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/build/gradleenv/1327086738/local/lib/python2.7/site-packages/apitools/base/py/transfer.py",
line 876, in InitializeUpload*22:01:22* return
self.StreamInChunks()*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/build/gradleenv/1327086738/local/lib/python2.7/site-packages/apitools/base/py/transfer.py",
line 988, in StreamInChunks*22:01:22*
additional_headers=additional_headers)*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/build/gradleenv/1327086738/local/lib/python2.7/site-packages/apitools/base/py/transfer.py",
line 939, in __StreamMedia*22:01:22*
self.RefreshResumableUploadState()*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/build/gradleenv/1327086738/local/lib/python2.7/site-packages/apitools/base/py/transfer.py",
line 841, in RefreshResumableUploadState*22:01:22*
self.stream.seek(self.progress)*22:01:22* File
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/filesystemio.py",
line 264, in seek*22:01:22* raise NotImplementedError*22:01:22*
NotImplementedError
It seems like this is a missing feature in Beam.
When an upload needs to resume and this is fails, apitools tries to seek to an
earlier point in the uploadable file
(https://github.com/google/apitools/blob/effc2e576427d8c876b1d64c79edcd98ab433074/apitools/base/py/transfer.py#L850),
however Beam filesystem does not support this
(https://github.com/apache/beam/blob/d02b859cb37e3c9f565785e93384905f1078b409/sdks/python/apache_beam/io/filesystemio.py#L264).
We can either update filesystemio to support this case, or add enable bundle
retrying capability for the direct runner.
> apache_beam.examples.wordcount_it_test.WordCountIT with DirectRunner failed
> ---------------------------------------------------------------------------
>
> Key: BEAM-6380
> URL: https://issues.apache.org/jira/browse/BEAM-6380
> Project: Beam
> Issue Type: Test
> Components: test-failures
> Reporter: Boyuan Zhang
> Assignee: Ahmet Altay
> Priority: Major
>
> wordcount test in :pythonPostCommit failed owing to RuntimeError:
> NotImplementedError [while running 'write/Write/WriteImpl/WriteBundles']
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/7001/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
