mapleFU commented on PR #41320: URL: https://github.com/apache/arrow/pull/41320#issuecomment-2074951956
See https://github.com/apache/arrow/issues/41317#issuecomment-2074947638 The root cause is that we didn't check the length. This can be detect by add the `ARROW_DCHECK_OK(table_.ValidateFull());` if debug enabled. Maybe we can solve this by more strict checkings in RecordReader ``` TableBatchReader::TableBatchReader(const Table& table) : owned_table_(nullptr), table_(table), column_data_(table.num_columns()), chunk_numbers_(table.num_columns(), 0), chunk_offsets_(table.num_columns(), 0), absolute_row_position_(0), max_chunksize_(std::numeric_limits<int64_t>::max()) { for (int i = 0; i < table.num_columns(); ++i) { column_data_[i] = table.column(i).get(); } ARROW_DCHECK_OK(table_.ValidateFull()); } TableBatchReader::TableBatchReader(std::shared_ptr<Table> table) : owned_table_(std::move(table)), table_(*owned_table_), column_data_(owned_table_->num_columns()), chunk_numbers_(owned_table_->num_columns(), 0), chunk_offsets_(owned_table_->num_columns(), 0), absolute_row_position_(0), max_chunksize_(std::numeric_limits<int64_t>::max()) { for (int i = 0; i < owned_table_->num_columns(); ++i) { column_data_[i] = owned_table_->column(i).get(); } ARROW_DCHECK_OK(table_.ValidateFull()); } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
