Amar3tto commented on issue #31354: URL: https://github.com/apache/beam/issues/31354#issuecomment-2170402703
There are some difficulties for making such `withFormatRecordOnFailureFunction()` equivalent for BigQuerySTORAGE_WRITE_API: - StorageApi has different implementations: - - BeamRow implementation that transforms original `ElementT` to `Row` - - Proto implementation that transforms original `ElementT` to `byte[]` - - Avro implementation that transforms original `ElementT` to `GenericRecord` - - TableRow implementation that transforms original `ElementT` to `TableRow` - _**After**_ that element to payload transformation, at the write records step, different `BigQueryStorageApiInsertError`s may occur (e.g. payload too large, etc.) - At this point (for any implementation), payload will be transformed to `TableRow` in order to output to `failedRowsTag`. - So it makes no sense to have `withFormatRecordOnFailureFunction()` from `ElementT` to `TableRow` (as it is implemented for STREAMING_INSERTS method), because at this step we only have a payload - There is a possibility to add `withFormatRecordOnFailureFunction()` from `TableRow` to `TableRow`, if it makes sense @sarinasij Could you please share an example how do you use `withFormatRecordOnFailureFunction()` with STREAMING_INSERTS method? I think it may help to clarify next steps -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@beam.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
