wjones127 opened a new issue, #34712: URL: https://github.com/apache/arrow/issues/34712
### Describe the enhancement requested We often parameterize things by number of rows, but what we would rather set the batch size in bytes. This is often the case when reading/writing files or IPC streams. One solution would be to provide utilities to estimate the average row size. For example, the Velox project file readers provide an `estimatedRowSize()` (although I'm not sure how often that is used): https://github.com/facebookincubator/velox/blob/33c40fda3a7654891c506bf23d078c0da0cd4f0d/velox/dwio/common/Reader.h#L71 This interface for the Parquet reader might be something like: ```cpp /// \brief Provides average bytes per row as in-memory Arrow data. /// /// \param sample_rows: if true, will read a sample of rows to estimate the average size of /// variable length columns Result<int64_t> EstimateDeserializedRowSize( parquet::arrow::FileReader file, std::vector<int64_t> column_indices, sample_rows = false); ``` Then would be used something like: ```cpp parquet::arrow::FileReader file = parquet::arrow::OpenFile("path/to/file"); std::vector<int64_t> columns = {1, 3, 5}; int64_t row_size = ARROW_RETURN_NOT_OK(EstimateDeserializedRowSize(file, columns)); // Configure Arrow-specific Parquet reader settings int64_t batch_size_bytes = 64 * 1024 * 1024; auto arrow_reader_props = parquet::ArrowReaderProperties(); arrow_reader_props.set_batch_size(batch_size_bytes / row_size); /// Then use the properties to get a RBR... ``` Similarly, when writing IPC we might want something like: ```cpp /// \brief Provides average bytes per for in serialized IPC message Result<int64_t> EstimateSerializedRowSize( const RecordBatch& batch, const IpcWriteOptions& write_option, ); ``` So we can use this when writing to a Flight stream: ```cpp std::shared_ptr<Table> table = ... int64_t row_size = EstimateSerializedRowSize(table->batch(0), write_options); int64_t batch_size_bytes = 10 * 1024 * 1024; auto reader = TableBatchReader(table.get()); reader.set_chunksize(batch_size_bytes / row_size); /// Pass batches to DoPut ``` ### Component(s) C++ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
