PHILO-HE commented on issue #10214: URL: https://github.com/apache/incubator-gluten/issues/10214#issuecomment-3584045961
Hi @ayushi-agarwal, Some users also reported that the deserialization time increased significantly in Gluten 1.4.0, compared with 1.2.1. They found the regression in TPC-DS. As suggested by @zhouyuan, with the patch below applied, the perf regression has been resolved according to their tests. I assume you are facing the same issue. Could you also have a try? - https://github.com/apache/incubator-gluten/pull/10922 Please note, you should also set `spark.gluten.sql.columnar.shuffle.readerBufferSize=0` to disable shuffle reader buffer, which is required to make this patch work. cc @marin-ma to see if you have any thoughts. It seems there are good cases and bad bases when using the buffer? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
