JkSelf commented on PR #12509: URL: https://github.com/apache/gluten/pull/12509#issuecomment-4976317237
@infvg @zhouyuan This issue was introduced by a recent [PR](https://github.com/facebookincubator/velox/pull/16998) where the code writes the batch before checking if the buffer size exceeds bytesInRowGroup, causing the row group to exceed the limit. Unlike Velox, the parquet-java reference implementation strictly ensures the row group size never exceeds bytesInRowGroup. Since Velox's current behavior deviates from this, I've opened a community [issue](https://github.com/facebookincubator/velox/issues/18123) to discuss how to define this in Velox. We should wait for that decision before making any changes here, as a fix on the Velox side would be preferred over patching it in Gluten. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
