ahmedabu98 commented on PR #32939: URL: https://github.com/apache/beam/pull/32939#issuecomment-2457454912
Agreed it's not the most ideal. I expected all these conversions to be taken care of behind the scenes by Iceberg's partitioning logic, but looks like for time types we need to it on our side instead. > recreating all records just to fit the types expected by the Iceberg partitioning logic We're not really recreating the records (i.e. not doing a fully copy). We're just creating an empty record and filling in the fields that we're partitioning on ([ref](https://github.com/apache/beam/pull/32939/files#diff-9112e1083b09d8f8ee32821c5c28e39965f3d65c02005ecb49f1f5182034f1c8R213-R214)). I found this to be the minimal implementation needed to make it work. In the average case, we should expect only a few fields to be populated, not the full record. I'm open to suggestions though! hope I'm missing something -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
