ahmedabu98 commented on issue #22961: URL: https://github.com/apache/beam/issues/22961#issuecomment-1298447876
Hey @pawelkopec, Looks like the error is thrown when temp files are finished writing at [`writer.close()`](https://github.com/apache/beam/blob/f46382110d78a6a8f03491ea2165a24dc1d5739c/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L276) and Beam tries to upload them to GCS. I'm not sure about the underlying reason for your HTTP errors but ideally this should be retried. Temp files are written to GCS and looking at the [uploading code in gcsio.py](https://github.com/apache/beam/blob/f46382110d78a6a8f03491ea2165a24dc1d5739c/sdks/python/apache_beam/io/gcp/gcsio.py#L744-L748), I see there is the `@retry.no_retries` decorator. @silviulica I saw your TODO tag there, do you have insight on this by any chance (I know this was committed a long time ago) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
