n3world commented on a change in pull request #11198:
URL: https://github.com/apache/arrow/pull/11198#discussion_r713182121
##########
File path: cpp/src/arrow/csv/reader_test.cc
##########
@@ -176,20 +177,21 @@ void
TestNestedParallelism(std::shared_ptr<internal::ThreadPool> thread_pool,
ASSERT_EQ(table->num_rows(), NROWS);
}
-void TestInvalidRowsSkipped(TableReaderFactory reader_factory, bool
rows_counted) {
+void TestInvalidRowsSkipped(TableReaderFactory reader_factory, bool async) {
const int NROWS = 1000;
const int INVALID_EVERY = 20;
const int NINVALID = 50;
auto opts = ParseOptions::Defaults();
- int num_invalid_rows = 0;
+ std::atomic<int> num_invalid_rows(0);
opts.invalid_row_handler = [&](const InvalidRow& row) {
++num_invalid_rows;
- if (rows_counted) {
+ if (async) {
+ // Row numbers are not counted in batches during async processing
+ EXPECT_EQ(-1, row.number);
+ } else {
// actual data starts at row #2 in the CSV "file"
EXPECT_EQ(num_invalid_rows * INVALID_EVERY + 1, row.number);
Review comment:
Yes. When it is not async the blocks are processed serially and so is
the callback. I can store the value locally just for atomic consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
