zeroshade commented on issue #374: URL: https://github.com/apache/arrow-go/issues/374#issuecomment-2867846354
Can you share the column metadata output that says it is 1GB? Is it perhaps referring to the uncompressed size? `RowGroupTotalBytesWritten()` will use the compressed size of the columns. Much of this will depend on the actual size of the records themselves that you are trying to write, since you only start a new record after you write something that exceeds your desired size. If you get close to the `maxRowGroupSize` and then write a very large record, you'll exceed the max by a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
