ahmedabu98 commented on PR #32879: URL: https://github.com/apache/beam/pull/32879#issuecomment-2427225036
@DanielMorales9 sorry but I did some experimenting and found it needed some more work, namely what was missing: - Refreshing our cached tables, so that we can use the updated partition spec when writing - Handling batches that contain files using old and new spec. Performing a single commit on all data files is no longer an option because commits are limited to one partition spec. However, performing multiple commits is not safe (if a bundle fails between commits, we may end up with data duplication). Instead, we can create a manifest file for each spec and append them all using one commit I fixed this up in the previous commit baba789323e351b6dab956db9cbb49c01ee142e6, can you take another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
