apanich commented on issue #38017: URL: https://github.com/apache/beam/issues/38017#issuecomment-4183128769
I think you get "Failed to copy Non partitioned table to Column partitioned table: not supported" when FILE_LOADS tries to load a large volume of data into your DAY-partitioned BigQuery table via additional copy to temp non-partitioned table. And this is probably what cases the error (on small data size no need to copy to temp non-partitioned table). So assuming you wan to keep using FILE_LOADS you would rather change triggering_frequency to lower value (sorry cannot give you right value, might need to play with it). One more alternative is to set withNumFileShards() - forcing it lower, maybe can help to keep direct copy. But I would not pursue this option - requires even more tuning to get good results. Finally please keep in mind that STORAGE_WRITE_API is the recommended way (while in some cases it is less cost effective, yet it is more reliable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
