Abacn commented on issue #24535: URL: https://github.com/apache/beam/issues/24535#issuecomment-1397889109
The bug can still occur when the number of files are greater than 10,000 then bigquery load jobs will be conducted in multiple partitions: https://github.com/apache/beam/blob/428ec97e30cc6587c2ae0f81d3ba44a8a9b34f93/sdks/python/apache_beam/io/gcp/bigquery_file_loads.py#L70 Did a test that decrease this number and force the write happens in multiple bigquery load jobs, still see dropping elements -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
