mosche commented on issue #23179: URL: https://github.com/apache/beam/issues/23179#issuecomment-1243650690
@bsikander Could you compare the DAG of your pipeline before and after the upgrade to help investigate. The increase in size you are seeing might suggest that the new version of Beam introduced an additional shuffle step that affects Parquet compression. Columnar formats offer powerful compression if data is adequately laid-out & ordered, but if changed things can look totally different. For instance, if a random shuffle is done before writing data, strategies such as `run length encoding` won't be able to compress the data much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
