reuvenlax commented on PR #17423: URL: https://github.com/apache/beam/pull/17423#issuecomment-1133474841
I'm not sure I understand. Retry _never_ makes sense here. If we failed to convert once, we will continue to fail to convert. The Storage Write API does not currently return per-row errors, however that is being added. However the plan is for it to only return errors for rows that cannot be inserted at all (e.g. the row is larger than allowed, a field does not match a schema constraint, etc.). A retry policy won't make sense there either, since such a failure means that the insert will continue to fail no matter how many times we retry it. The failedInsertRetryPolicy made sense for the old InsertAll API, since that API would return back errors that were retryable, and the user needed to specify sometimes whether to retry or not (though that API was always incomplete, since users often wanted to specify a maximum number of retries which was not supported). Here I don't think it really makes sense. On Fri, May 20, 2022 at 7:47 PM Chamikara Jayalath ***@***.***> wrote: > Can we build a retry policy that only include "SchemaConversionExceptions" > by default ? > > Also, can we implement something similar to the failedInsertRetryPolicy > for the Storage Write API ? > > https://github.com/apache/beam/blob/0c9cf43a7edae2e2a2622a8f4241b64a638121bb/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L2340 > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/beam/pull/17423#issuecomment-1133468862>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFAYJVOFEXS5RKSF4OQDQ63VLAQBLANCNFSM5T6AVIFA> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
