wjones127 opened a new issue, #34712:
URL: https://github.com/apache/arrow/issues/34712

   ### Describe the enhancement requested
   
   We often parameterize things by number of rows, but what we would rather set 
the batch size in bytes. This is often the case when reading/writing files or 
IPC streams. One solution would be to provide utilities to estimate the average 
row size. For example, the Velox project file readers provide an 
`estimatedRowSize()` (although I'm not sure how often that is used):
   
   
https://github.com/facebookincubator/velox/blob/33c40fda3a7654891c506bf23d078c0da0cd4f0d/velox/dwio/common/Reader.h#L71
   
   This interface for the Parquet reader might be something like:
   
   ```cpp
   /// \brief Provides average bytes per row as in-memory Arrow data.
   ///
   /// \param sample_rows: if true, will read a sample of rows to estimate the 
average size of
   /// variable length columns
   Result<int64_t> EstimateDeserializedRowSize(
     parquet::arrow::FileReader file,
     std::vector<int64_t> column_indices,
     sample_rows = false);
   ```
   
   Then would be used something like:
   
   ```cpp
   parquet::arrow::FileReader file = parquet::arrow::OpenFile("path/to/file");
   std::vector<int64_t> columns = {1, 3, 5};
   int64_t row_size = ARROW_RETURN_NOT_OK(EstimateDeserializedRowSize(file, 
columns));
   
   // Configure Arrow-specific Parquet reader settings
   int64_t batch_size_bytes = 64 * 1024 * 1024;
   auto arrow_reader_props = parquet::ArrowReaderProperties();
   arrow_reader_props.set_batch_size(batch_size_bytes / row_size);
   
   /// Then use the properties to get a RBR...
   ```
   
   
   Similarly, when writing IPC we might want something like:
   
   ```cpp
   /// \brief Provides average bytes per for in serialized IPC message
   Result<int64_t> EstimateSerializedRowSize(
     const RecordBatch& batch,
     const IpcWriteOptions& write_option,
   );
   ```
   
   So we can use this when writing to a Flight stream:
   
   ```cpp
   std::shared_ptr<Table> table = ...
   int64_t row_size = EstimateSerializedRowSize(table->batch(0), write_options);
   
   int64_t batch_size_bytes = 10 * 1024 * 1024;
   auto reader = TableBatchReader(table.get());
   reader.set_chunksize(batch_size_bytes / row_size);
   
   /// Pass batches to DoPut
   ```
   
   ### Component(s)
   
   C++


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to