carlpayne commented on issue #24090: URL: https://github.com/apache/beam/issues/24090#issuecomment-1322222261
@reuvenlax The main examples we've encountered are schema-mismatches (e.g. missing a required field, as per https://github.com/apache/beam/issues/23291). In this case, we need to manually (or automatically, if https://github.com/apache/beam/issues/24063 becomes possible) update the table schema and retry. While it would be possible to do this via DLQ in a separate BigQuery table, this would be inconsistent with many of our other applications and re-stream processes where DLQ is always via Kafka (we use other stream-processing tools such as Flink where Kafka is the DLQ). We also prefer to keep the "raw" data for replay purposes, rather than the converted TableRow, just to rule out an issue with the raw-to-TableRow conversion process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
