marklit commented on issue #15220: URL: https://github.com/apache/arrow/issues/15220#issuecomment-1378222568
I talked about writing PQ out into separate parallel files in my [blog post](https://tech.marksblogg.com/postgresql-to-bigquery.html) yesterday, it does speed things up a lot if the CPU core count of your system is high enough. I'd need to dig into CH's command line switches to see what exactly is possible with tuning their PQ writer and then seeing if these settings have a material impact on their perf and of your project as well. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
