bkietz commented on code in PR #35320: URL: https://github.com/apache/arrow/pull/35320#discussion_r1179166745
########## docs/source/cpp/acero/overview.rst: ########## @@ -0,0 +1,262 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp +.. cpp:namespace:: arrow::acero + +============== +Acero Overview +============== + +This page gives an overview of the basic Acero concepts and helps distinguish Acero +from other modules in the Arrow code base. It's intended for users, developers, +potential contributors, and for those that would like to extend Acero, either for +research or for business use. This page assumes the reader is already familiar with +core Arrow concepts. This page does not expect any existing knowledge in relational +algebra. + +What is Acero? +============== + +Acero is a C++ library that can be used to analyze large (potentially infinite) streams +of data. Acero allows computation to be expressed as an "execution plan" (:class:`ExecPlan`). +An execution plan takes in zero or more streams of input data and emits a single +stream of output data. The plan describes how the data will be transformed as it +passes through. For example, a plan might: + + * Merge two streams of data using a common column + * Create additional columns by evaluating expressions against the existing columns + * Consume a stream of data by writing it to disk in a partitioned layout + +.. image:: simple_graph.svg + :alt: A sample execution plan that joins three streams of data and writes to disk + +Acero is not... +--------------- + +A Library for Data Scientists +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Acero is not intended to be used directly by data scientists. It is expected that +end users will typically be using some kind of frontend. For example, Pandas, Ibis, +or SQL. The API for Acero is focused around capabilities and available algorithms. +However, such users may be intersted in knowing more about how Acero works so that +they can better understand how the backend processing for their libraries operates. + +A Database +^^^^^^^^^^ + +A database (or DBMS) is typically a much more expansive application and often packaged +as a standalone service. Acero could be a component in a database (most databases have +some kind of execution engine) or could be a component in some other data processing +application that hardly resembles a database. Acero does not concern itself with +user management, external communication, isolation, durability, or consistency. In +addition, Acero is focused primarily on the read path, and the write utilities lack +any sort of transaction support. + +An Optimizer +^^^^^^^^^^^^ + +Acero does not have an SQL parser. It does not have a query planner. It does not have +any sort of optimizer. Acero expects to be given very detailed and low-level instructions +on how to manipulate data and then it will perform that manipulation exactly as described. + +Creating the best execution plan is very hard. Small details can have a big impact on +performance. We do think optimizers are important and we hope that tools will emerge +someday which can provide these capabilities using standards such as Substrait. Review Comment: ```suggestion Creating the best execution plan is very hard. Small details can have a big impact on performance. We do think an optimizer is important but we believe it should be implemented independent of acero, hopefully in a composable way through standards such as Substrait so that any backend could leverage it. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
