iChauster commented on code in PR #13314: URL: https://github.com/apache/arrow/pull/13314#discussion_r890478754
########## cpp/src/arrow/compute/exec/project_benchmark.cc: ########## @@ -0,0 +1,104 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "benchmark/benchmark.h" + +#include "arrow/compute/cast.h" +#include "arrow/compute/exec.h" +#include "arrow/compute/exec/expression.h" +#include "arrow/compute/exec/options.h" +#include "arrow/compute/exec/test_util.h" +#include "arrow/dataset/partition.h" +#include "arrow/testing/future_util.h" +#include "arrow/testing/generator.h" +#include "arrow/testing/gtest_util.h" +#include "arrow/type.h" + +namespace arrow { +namespace compute { + +static constexpr int64_t total_batch_size = 1e6; + +void SetArgs(benchmark::internal::Benchmark* bench) { + for (auto batch_size = 1; batch_size <= 1e6; batch_size *= 10) { + auto num_batches = total_batch_size / batch_size; + bench->ArgNames({"batch_size, num_batches"}) + ->Args({batch_size, num_batches}) + ->UseRealTime(); + } +} + +/* +should be able to vary +1. batch size +2. expression complexity +*/ +static void ExecuteScalarProjectionOverhead(benchmark::State& state, Expression expr) { + const auto batch_size = static_cast<int32_t>(state.range(0)); + const auto num_batches = static_cast<int32_t>(state.range(1)); + + auto data = MakeRandomBatches(schema({field("i64", int64()), field("bool", boolean())}), + num_batches, batch_size); + ExecContext ctx; + + for (auto _ : state) { + ASSERT_OK_AND_ASSIGN(auto plan, ExecPlan::Make()); + // CountOptions options(CountOptions::ONLY_VALID); + AsyncGenerator<util::optional<ExecBatch>> sink_gen; + ASSERT_OK(Declaration::Sequence( + { + {"source", + SourceNodeOptions{data.schema, + data.gen(/*parallel=*/false, /*slow=*/false)}, + "custom_source_label"}, + {"project", ProjectNodeOptions{{ + field_ref("bool"), + expr, + }}}, + {"sink", SinkNodeOptions{&sink_gen}, "custom_sink_label"}, + }) + .AddToPlan(plan.get())); + ASSERT_FINISHES_OK(StartAndCollect(plan.get(), sink_gen)); Review Comment: I had a look around the codebase and haven't yet seen any manual implementation like this -- let me know if something like this is right: Should I write a `source_node.h` and a `project_node.h`? I haven't yet seen examples of constructing these types of nodes manually. **EDIT: Using Factories now.** Once I am able to create a project_node, 1. I assume I can just have some dummy plan and dummy input source. 2. From there, I need to generate `ExecBatch` (a vector of them) -- is this possible to be done as a cast / conversion from `MakeRandomBatches`, or does it have to pass through `source_node`? **EDIT: Yes.** 3. Call `InputReceived` with `source_node` and one of the batches, and call `InputFinished` with the `source_node` and number of batches 4. Wait for `finished()` Let me know if I'm on the right track, or if there anywhere I can look for an easier way to do this. Thanks again for the help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
