FelixYBW commented on PR #5952: URL: https://github.com/apache/incubator-gluten/pull/5952#issuecomment-2254878630
> ow operation is completed in one go, which consumes a lot of memory and causes the program OOM. In this commit, I modified the C2 @XinShuoWang what's your case to have large batch size? My issue can be fixed by reduce spark.gluten.sql.columnar.maxBatchSize and spark.sql.parquet.columnarReaderBatchSize -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
