marin-ma commented on PR #7897: URL: https://github.com/apache/incubator-gluten/pull/7897#issuecomment-2475208389
> Does vanilla spark have the same issue? @FelixYBW Spark used `BufferedInputStream` with a fixed size of 8192. https://github.com/apache/spark/blob/branch-3.2/sql/core/src/main/scala/org/apache/spark/sql/execution/UnsafeRowSerializer.scala#L102 I've tested with 16k buffer size for this patch, but there's not much perf gain. The performance in the PR description uses 1M buffer size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
