westonpace commented on code in PR #15183:
URL: https://github.com/apache/arrow/pull/15183#discussion_r1063021401
##########
cpp/src/arrow/compute/exec/plan_test.cc:
##########
@@ -344,6 +344,47 @@ void TestSourceSink(
Finishes(ResultWith(UnorderedElementsAreArray(exp_batches.batches))));
}
+void TestRecordBatchReaderSourceSink(
+ std::function<Result<std::shared_ptr<RecordBatchReader>>(const
BatchesWithSchema&)>
+ to_reader) {
+ ASSERT_OK_AND_ASSIGN(auto executor, arrow::internal::ThreadPool::Make(1));
+ ExecContext exec_context(default_memory_pool(), executor.get());
+ ASSERT_OK_AND_ASSIGN(auto plan, ExecPlan::Make(exec_context));
+ AsyncGenerator<std::optional<ExecBatch>> sink_gen;
+
+ auto exp_batches = MakeBasicBatches();
+ ASSERT_OK_AND_ASSIGN(std::shared_ptr<RecordBatchReader> reader,
to_reader(exp_batches));
+
+ ASSERT_OK(Declaration::Sequence(
+ {
+ {"record_batch_reader_source",
+ RecordBatchReaderSourceNodeOptions{exp_batches.schema,
reader}},
+ {"sink", SinkNodeOptions{&sink_gen}},
+ })
+ .AddToPlan(plan.get()));
+
+ ASSERT_THAT(StartAndCollect(plan.get(), sink_gen),
+
Finishes(ResultWith(UnorderedElementsAreArray(exp_batches.batches))));
+}
Review Comment:
> Are you referring to IOThreadPool being used as the default instead of
NULLPTR?
What you had before might actually have been correct (I thought you were
using `executor` in the options but on closer examination it was only being
used in the exec_ctx). I was saying the most common case is
`RecordBatchReaderSourceNodeOptions options(reader)` where the executor is not
specified and is left to be the default. We should have our test verify that
this case works correctly. At the very least we should make sure we can run
such a plan without crashing. Long term we should check, after we ran the
plan, to ensure that there were tasks submitted to the I/O thread pool.
This kind of test should not need to create a thread pool at all. We should
only need to create a thread pool when testing a custom executor in the node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
