westonpace commented on a change in pull request #10061:
URL: https://github.com/apache/arrow/pull/10061#discussion_r616021323
##########
File path: cpp/src/arrow/util/async_generator.h
##########
@@ -1475,11 +1475,44 @@ static Result<AsyncGenerator<T>>
MakeBackgroundGenerator(
return BackgroundGenerator<T>(std::move(iterator), io_executor, max_q,
q_restart);
}
-/// \see MakeGeneratorIterator
-template <typename T>
+template <typename GeneratorFactory,
+ typename Generator =
internal::call_traits::return_type<GeneratorFactory>,
+ typename FT = typename Generator::result_type,
+ typename T = typename FT::ValueType>
class GeneratorIterator {
public:
- explicit GeneratorIterator(AsyncGenerator<T> source) :
source_(std::move(source)) {}
+ explicit GeneratorIterator(GeneratorFactory factory)
+ : proxy_executor_(new internal::ProxyExecutor()) {
+ source_ = factory(proxy_executor_.get());
+ }
+
+ Result<T> Next() {
+ return internal::SerialExecutor::RunInSerialExecutor<T>(
+ [this](internal::Executor* executor) {
+ proxy_executor_->target = executor;
+ return source_();
+ });
+ }
+
+ private:
+ AsyncGenerator<T> source_;
+ std::unique_ptr<internal::ProxyExecutor> proxy_executor_;
+};
+
+template <typename GeneratorFactory,
+ typename Generator =
internal::call_traits::return_type<GeneratorFactory>,
+ typename FT = typename Generator::result_type,
+ typename T = typename FT::ValueType>
+Result<Iterator<T>> MakeGeneratorIterator(GeneratorFactory factory) {
Review comment:
Yikes, I'm glad you mentioned this. It looks like I had rebased on top
of ARROW-12412 because I had planned at using that here until it ended up
failing. I'm sorry I didn't catch this earlier. The problem ends up being
that readahead callbacks get completed between pulls when the proxy iterator
has no valid value. I've cut that code out now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
