zeroshade commented on code in PR #36163:
URL: https://github.com/apache/arrow/pull/36163#discussion_r1234553152
##########
go/parquet/pqarrow/file_writer.go:
##########
@@ -134,6 +134,13 @@ func (fw *FileWriter) RowGroupTotalBytesWritten() int64 {
return 0
}
+// WriteBuffered allows to write records and decide where to break your row
group
+// based on the TotalBytesWritten rather than on the max row group len.
Review Comment:
TotalBytesWritten will still break row groups on the max row group len, not
TotalBytesWritten. It does, however, allow users to use TotalBytesWritten to
decide when to break row groups themselves by checking `TotalBytesWritten` and
calling `NewBuffereredRowGroup` themselves.
##########
go/parquet/pqarrow/file_writer.go:
##########
@@ -134,6 +134,13 @@ func (fw *FileWriter) RowGroupTotalBytesWritten() int64 {
return 0
}
+// WriteBuffered allows to write records and decide where to break your row
group
+// based on the TotalBytesWritten rather than on the max row group len.
Review Comment:
`WriteBuffered` will still break row groups on the max row group len, not
TotalBytesWritten. It does, however, allow users to use TotalBytesWritten to
decide when to break row groups themselves by checking `TotalBytesWritten` and
calling `NewBuffereredRowGroup` themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
