wjones127 commented on code in PR #13859: URL: https://github.com/apache/arrow/pull/13859#discussion_r960063500
########## docs/source/cpp/getting_started.rst: ########## @@ -18,28 +18,13 @@ .. default-domain:: cpp .. highlight:: cpp -User Guide -========== +Getting Started +=============== Review Comment: It would be nice to have a prose intro on this page. Maybe something that teases the capabilities of Arrow, and set expectations for the level of detail to find in this section of the docs. ########## docs/source/cpp/tutorials/compute_tutorial.rst: ########## @@ -0,0 +1,342 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +============= +Arrow Compute +============= + +Apache Arrow provides compute functions to facilitate efficient and +portable data processing. In this article, you will use Arrow’s compute +functionality to: + +1. Calculate a sum over a column + +2. Calculate element-wise sums over two columns + +3. Search for a value in a column + +Pre-requisites +--------------- + +Before continuing, make sure you have: + +1. An Arrow installation + +2. An understanding of basic Arrow data structures from <basic data structures> + +Setup +----- + +Before running some computations, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. ``A main()`` is needed to glue things together. + +3. We need data to play with. + +Includes +^^^^^^^^ + +Before writing C++ code, we need some includes. We'll get ``iostream`` for output, then import Arrow's +compute functionality for each file type we'll work with in this article: Review Comment: ```suggestion compute functionality: ``` ########## docs/source/cpp/tutorials/basic_arrow.rst: ########## @@ -0,0 +1,283 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +=========================== +Basic Arrow Data Structures +=========================== + +Apache Arrow provides fundamental data structures for representing data: +:class:`Array`, :class:`ChunkedArray`, :class:`RecordBatch`, and :class:`Table`. +This article shows how to construct these data structures from primitive +data types; specifically, we will work with integers of varying size +representing days, months, and years. We will use them to create the following data structures: + +#. Arrow :class:`Arrays <Array>` +#. :class:`ChunkedArrays<ChunkedArray>` +#. :class:`RecordBatch`, from :class:`Arrays <Array>` +#. :class:`Table`, from :class:`ChunkedArrays<ChunkedArray>` + +Pre-requisites +-------------- +Before continuing, make sure you have: + +#. An Arrow installation +#. Understanding of how to use basic C++ data structures +#. Understanding of basic C++ data types + + +Setup +----- + +Before trying out Arrow, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. ``A main()`` is needed to glue things together. + +Includes +^^^^^^^^ + +First, as ever, we need some includes. We'll get ``iostream`` for output, then import Arrow's basic +functionality from ``api.h``, like so: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +Next, we need a ``main()`` – a common pattern with Arrow looks like the +following: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +This allows us to easily use Arrow’s error-handling macros, which will +return back to ``main()`` with a :class:`arrow::Status` object if a failure occurs – and +this ``main()`` will report the error. Note that this means Arrow never +raises exceptions, instead relying upon returning :class:`Status`. For more on +that, read here: <LINK TO CONCEPTUAL OVERVIEW> + +To accompany this ``main()``, we have a ``RunMain()`` from which any :class:`Status` +objects can return – this is where we’ll write the rest of the program: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: RunMain Start) + :end-before: (Doc section: RunMain Start) + + +Making an Arrow Array +--------------------- + +Building int8 Arrays +^^^^^^^^^^^^^^^^^^^^ + +Given that we have some data in standard C++ arrays, and want to use Arrow, we need to move +the data from said arrays into Arrow arrays. We still guarantee contiguity of memory in an +:class:`Array`, so no worries about a performance loss when using :class:`Array` vs C++ arrays. +The easiest way to construct an :class:`Array` uses an :class:`ArrayBuilder`. <RST +NOTE NEAR HERE: for more technical details, check out…> Review Comment: One detail to mention and link to is that we are building `Array` classes, which is a type that is agnostic to the array data type (int8). But this array can be casted to the concrete `arrow::Int8Array` and (somehow) that will let you access the int8 values manually, if you wanted to. ########## docs/source/cpp/tutorials/io_tutorial.rst: ########## @@ -0,0 +1,404 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +============== +Arrow File I/O +============== + +Apache Arrow provides file I/O functions to facilitate use of Arrow from +the start to end of an application. In this article, you will: + +1. Read an Arrow file into a :class:`RecordBatch` and write it back out afterwards + +2. Read a CSV file into a :class:`Table` and write it back out afterwards + +3. Read a Parquet file into a :class:`Table` and write it back out afterwards + +Pre-requisites +--------------- + +Before continuing, make sure you have: + +1. An Arrow installation + +2. An understanding of basic Arrow data structures from <the preceding article> + +3. A directory to run the final application in – this program will generate some files, so be prepared for that. + +Setup +----- + +Before writing out some file I/O, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. A ``main()`` is needed to glue things together. + +3. We need files to play with. + +Includes +^^^^^^^^ + +Before writing C++ code, we need some includes. We'll get ``iostream`` for output, then import Arrow's +I/O functionality for each file type we'll work with in this article: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +For our glue, we’ll use the ``main()`` pattern from the previous tutorial on +data structures: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +Which, like when we used it before, is paired with a ``RunMain()``: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: RunMain) + :end-before: (Doc section: RunMain) + +Generating Files for Reading +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We need some files to actually play with. In practice, you’ll likely +have some input for your own application. Here, however, we want to +explore doing I/O for the sake of it, so let’s generate some files to make +this easy to follow. To create those, we’ll define a helper function +that we’ll run first. Feel free to read through this, but the concepts +used will be explained later in this article. Note that we’re using the +day/month/year data from the previous tutorial. For now, just copy the +function in: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: GenInitialFile) + :end-before: (Doc section: GenInitialFile) + +To get the files for the rest of your code to function, make sure to +call ``GenInitialFile()`` as the very first line in ``RunMain()`` to initialize +the environment: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Gen Files) + :end-before: (Doc section: Gen Files) + +I/O with Arrow Files +-------------------- + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Bind file to :class:`ipc::RecordBatchFileReader` + + c. Read file to :class:`RecordBatch` + +2. Writing a file + + a. Get a :class:`io::FileOutputStream` + + b. Write to file from :class:`RecordBatch` + +Opening a File +^^^^^^^^^^^^^^ + +To actually read a file, we need to get some sort of way to point to it. +In Arrow, that means we’re going to get a :class:`io::ReadableFile` object – much +like an :class:`ArrayBuilder` can clear and make new arrays, we can reassign this +to new files, so we’ll use this instance throughout the examples: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: ReadableFile Definition) + :end-before: (Doc section: ReadableFile Definition) + +A :class:`io::ReadableFile` does little alone – we actually have it bind to a file +with :func:`io::ReadableFile::Open`. You can read more on that and its arguments at <link>. For +our purposes here, the default arguments suffice: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow ReadableFile Open) + :end-before: (Doc section: Arrow ReadableFile Open) + +Opening an Arrow file Reader +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An :class:`io::ReadableFile` is too generic to offer all functionality to read an Arrow file. +We need to use it to get an :class:`ipc::RecordBatchFileReader` object. This object implements +all the logic needed to read an Arrow file with correct formatting. We get one through +:func:`ipc::RecordBatchFileReader::Open`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read Open) + :end-before: (Doc section: Arrow Read Open) + +Reading an Open Arrow File to RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We have to use a :class:`RecordBatch` to read an Arrow file, so we’ll get a +:class:`RecordBatch`. Once we have that, we can actually read the file. Arrow +files can have multiple :class:`RecordBatches <RecordBatch>`, so we must pass an index. This +file only has one, so pass 0: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read) + :end-before: (Doc section: Arrow Read) + +Prepare a FileOutputStream +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For output, we need a :class:`io::FileOutputStream`. Just like our :class:`io::ReadableFile`, +we’ll be reusing this, so be ready for that. We open files the same way +as when reading: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Write Open) + :end-before: (Doc section: Arrow Write Open) + +Write Arrow File from RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, we grab our :class:`RecordBatch` we read into previously, and use it, along +with our target file, to create a :class:`ipc::RecordBatchWriter`. The +:class:`ipc::RecordBatchWriter` needs two things: + +1. the target file + +2. the :class:`Schema` for our :class:`RecordBatch` (in case we need to write more :class:`RecordBatches <RecordBatch>` of the same format.) + +The :class:`Schema` comes from our existing :class:`RecordBatch`, and the target file is +just a name – in this case, test_out.arrow. + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Writer) + :end-before: (Doc section: Arrow Writer) + +We can just call :func:`ipc::RecordBatchWriter::WriteRecordBatch` with our :class:`RecordBatch` to fill up our +file: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Write) + :end-before: (Doc section: Arrow Write) + +For IPC in particular, the writer has to be closed, so do that: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Close) + :end-before: (Doc section: Arrow Close) + +Now we’ve read and written an IPC file! + +I/O with CSV +------------ + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Prepare Table + + c. Read File using :class:`csv::TableReader` + +2. Writing a file + + a. Get a :class:`io::FileOutputStream` + + b. Write to file from :class:`Table` + +Opening a CSV File +^^^^^^^^^^^^^^^^^^ + +For a CSV file, we need to open a :class:`io::ReadableFile`, just like an Arrow file, +and reuse our :class:`io::ReadableFile` object from before to do so: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Read Open) + :end-before: (Doc section: CSV Read Open) + +Preparing a Table +^^^^^^^^^^^^^^^^^ + +CSV can be read into a :class:`Table`, so declare a pointer to a :class:`Table`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Table Declare) + :end-before: (Doc section: CSV Table Declare) + +Read a CSV File to Table +^^^^^^^^^^^^^^^^^^^^^^^^ + +The CSV reader has option structs which need to be passed – luckily, +there are defaults for these which we can pass directly. For reference +on the other options, go here: <link>. This CSV is a standard CSV +without any special delimiters and is small, so we can make our reader +with defaults: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Reader Make) + :end-before: (Doc section: CSV Reader Make) + +With the CSV reader primed, we can use its :func:`csv::TableReader::Read` method to fill our +:class:`Table`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Read) + :end-before: (Doc section: CSV Read) + +Write a CSV File from Table +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +CSV writing to :class:`Table` looks exactly like IPC writing to :class:`RecordBatch`, +except with our :class:`Table`, and using :func:`ipc::RecordBatchWriter::WriteTable` instead of Review Comment: The `ipc` namespacing is unfortunate; I instinctively though this was wrong (`ipc` namespace from `csv`!?), but it isn't! ########## docs/source/cpp/getting_started.rst: ########## @@ -18,28 +18,16 @@ .. default-domain:: cpp .. highlight:: cpp -User Guide -========== +Getting Started +=============== .. toctree:: - overview - conventions build_system Review Comment: I know you didn't introduce this, but would you mind fixing the bullets on this page, which are indented in the RST and thus are within block quotes on the rendered page? ########## docs/source/cpp/tutorials/io_tutorial.rst: ########## @@ -0,0 +1,404 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +============== +Arrow File I/O +============== + +Apache Arrow provides file I/O functions to facilitate use of Arrow from +the start to end of an application. In this article, you will: + +1. Read an Arrow file into a :class:`RecordBatch` and write it back out afterwards + +2. Read a CSV file into a :class:`Table` and write it back out afterwards + +3. Read a Parquet file into a :class:`Table` and write it back out afterwards + +Pre-requisites +--------------- + +Before continuing, make sure you have: + +1. An Arrow installation + +2. An understanding of basic Arrow data structures from <the preceding article> + +3. A directory to run the final application in – this program will generate some files, so be prepared for that. + +Setup +----- + +Before writing out some file I/O, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. A ``main()`` is needed to glue things together. + +3. We need files to play with. + +Includes +^^^^^^^^ + +Before writing C++ code, we need some includes. We'll get ``iostream`` for output, then import Arrow's +I/O functionality for each file type we'll work with in this article: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +For our glue, we’ll use the ``main()`` pattern from the previous tutorial on +data structures: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +Which, like when we used it before, is paired with a ``RunMain()``: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: RunMain) + :end-before: (Doc section: RunMain) + +Generating Files for Reading +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We need some files to actually play with. In practice, you’ll likely +have some input for your own application. Here, however, we want to +explore doing I/O for the sake of it, so let’s generate some files to make +this easy to follow. To create those, we’ll define a helper function +that we’ll run first. Feel free to read through this, but the concepts +used will be explained later in this article. Note that we’re using the +day/month/year data from the previous tutorial. For now, just copy the +function in: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: GenInitialFile) + :end-before: (Doc section: GenInitialFile) + +To get the files for the rest of your code to function, make sure to +call ``GenInitialFile()`` as the very first line in ``RunMain()`` to initialize +the environment: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Gen Files) + :end-before: (Doc section: Gen Files) + +I/O with Arrow Files +-------------------- + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Bind file to :class:`ipc::RecordBatchFileReader` + + c. Read file to :class:`RecordBatch` + +2. Writing a file + + a. Get a :class:`io::FileOutputStream` + + b. Write to file from :class:`RecordBatch` + +Opening a File +^^^^^^^^^^^^^^ + +To actually read a file, we need to get some sort of way to point to it. +In Arrow, that means we’re going to get a :class:`io::ReadableFile` object – much +like an :class:`ArrayBuilder` can clear and make new arrays, we can reassign this +to new files, so we’ll use this instance throughout the examples: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: ReadableFile Definition) + :end-before: (Doc section: ReadableFile Definition) + +A :class:`io::ReadableFile` does little alone – we actually have it bind to a file +with :func:`io::ReadableFile::Open`. You can read more on that and its arguments at <link>. For +our purposes here, the default arguments suffice: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow ReadableFile Open) + :end-before: (Doc section: Arrow ReadableFile Open) + +Opening an Arrow file Reader +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An :class:`io::ReadableFile` is too generic to offer all functionality to read an Arrow file. +We need to use it to get an :class:`ipc::RecordBatchFileReader` object. This object implements +all the logic needed to read an Arrow file with correct formatting. We get one through +:func:`ipc::RecordBatchFileReader::Open`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read Open) + :end-before: (Doc section: Arrow Read Open) + +Reading an Open Arrow File to RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We have to use a :class:`RecordBatch` to read an Arrow file, so we’ll get a +:class:`RecordBatch`. Once we have that, we can actually read the file. Arrow +files can have multiple :class:`RecordBatches <RecordBatch>`, so we must pass an index. This +file only has one, so pass 0: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read) + :end-before: (Doc section: Arrow Read) + +Prepare a FileOutputStream +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For output, we need a :class:`io::FileOutputStream`. Just like our :class:`io::ReadableFile`, +we’ll be reusing this, so be ready for that. We open files the same way +as when reading: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Write Open) + :end-before: (Doc section: Arrow Write Open) + +Write Arrow File from RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, we grab our :class:`RecordBatch` we read into previously, and use it, along +with our target file, to create a :class:`ipc::RecordBatchWriter`. The +:class:`ipc::RecordBatchWriter` needs two things: + +1. the target file + +2. the :class:`Schema` for our :class:`RecordBatch` (in case we need to write more :class:`RecordBatches <RecordBatch>` of the same format.) + +The :class:`Schema` comes from our existing :class:`RecordBatch`, and the target file is +just a name – in this case, test_out.arrow. Review Comment: ```suggestion The :class:`Schema` comes from our existing :class:`RecordBatch` and the target file is the output stream we just created. ``` ########## docs/source/cpp/tutorials/basic_arrow.rst: ########## @@ -0,0 +1,283 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +=========================== +Basic Arrow Data Structures +=========================== + +Apache Arrow provides fundamental data structures for representing data: +:class:`Array`, :class:`ChunkedArray`, :class:`RecordBatch`, and :class:`Table`. +This article shows how to construct these data structures from primitive +data types; specifically, we will work with integers of varying size +representing days, months, and years. We will use them to create the following data structures: + +#. Arrow :class:`Arrays <Array>` +#. :class:`ChunkedArrays<ChunkedArray>` +#. :class:`RecordBatch`, from :class:`Arrays <Array>` +#. :class:`Table`, from :class:`ChunkedArrays<ChunkedArray>` + +Pre-requisites +-------------- +Before continuing, make sure you have: + +#. An Arrow installation +#. Understanding of how to use basic C++ data structures +#. Understanding of basic C++ data types + + +Setup +----- + +Before trying out Arrow, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. ``A main()`` is needed to glue things together. + +Includes +^^^^^^^^ + +First, as ever, we need some includes. We'll get ``iostream`` for output, then import Arrow's basic +functionality from ``api.h``, like so: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +Next, we need a ``main()`` – a common pattern with Arrow looks like the +following: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +This allows us to easily use Arrow’s error-handling macros, which will +return back to ``main()`` with a :class:`arrow::Status` object if a failure occurs – and +this ``main()`` will report the error. Note that this means Arrow never +raises exceptions, instead relying upon returning :class:`Status`. For more on +that, read here: <LINK TO CONCEPTUAL OVERVIEW> + +To accompany this ``main()``, we have a ``RunMain()`` from which any :class:`Status` +objects can return – this is where we’ll write the rest of the program: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: RunMain Start) + :end-before: (Doc section: RunMain Start) + + +Making an Arrow Array +--------------------- + +Building int8 Arrays +^^^^^^^^^^^^^^^^^^^^ + +Given that we have some data in standard C++ arrays, and want to use Arrow, we need to move +the data from said arrays into Arrow arrays. We still guarantee contiguity of memory in an +:class:`Array`, so no worries about a performance loss when using :class:`Array` vs C++ arrays. +The easiest way to construct an :class:`Array` uses an :class:`ArrayBuilder`. <RST +NOTE NEAR HERE: for more technical details, check out…> + +The following code initializes an :class:`ArrayBuilder` for an :class:`Array` that will hold 8 bit +integers. Specifically, it uses the ``AppendValues()`` method, present in concrete +:class:`arrow::ArrayBuilder` subclasses, to fill the :class:`ArrayBuilder` with the +contents of a standard C++ array. Note the use of :c:macro:`ARROW_RETURN_NOT_OK`. +If ``AppendValues()`` fails, this macro will return to ``main()``, which will +print out the meaning of the failure. + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: int8builder 1 Append) + :end-before: (Doc section: int8builder 1 Append) + +Given an :class:`ArrayBuilder` has the values we want in our :class:`Array`, we can use +:func:`ArrayBuilder::Finish` to output the final structure to an :class:`Array` – specifically, +we output to a ``std::shared_ptr<arrow::Array>``. Note the use of :c:macro:`ARROW_ASSIGN_OR_RAISE` +in the following code. :func:`~ArrayBuilder::Finish` outputs a :class:`arrow::Result` object, which :c:macro:`ARROW_ASSIGN_OR_RAISE` +can process. If the method fails, it will return to ``main()`` with a :class:`Status` +that will explain what went wrong. If it succeeds, then it will assign +the final output to the left-hand variable. + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: int8builder 1 Finish) + :end-before: (Doc section: int8builder 1 Finish) + +As soon as :class:`ArrayBuilder` has had its :func:`Finish <ArrayBuilder::Finish>` method called, its state resets, so +it can be used again, as if it was fresh. Thus, we repeat the process above for our second array: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: int8builder 2) + :end-before: (Doc section: int8builder 2) + +Building int16 Arrays +^^^^^^^^^^^^^^^^^^^^^ + +An :class:`ArrayBuilder` has its type specified at the time of declaration. +Once this is done, it cannot have its type changed. We have to make a new one when we switch to year data, which +requires a 16-bit integer at the minimum. Of course, there’s an :class:`ArrayBuilder` for that. +It uses the exact same methods, but with the new data type: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: int16builder) + :end-before: (Doc section: int16builder) + +Now, we have three Arrow :class:`Arrays <arrow::Array>`, with some variance in type. + +Making a RecordBatch +-------------------- + +A columnar data format only really comes into play when you have a table. +So, let’s make one. The first kind we’ll make is the :class:`RecordBatch` – this +uses :class:`Arrays <Array>` internally, which means all data will be contiguous within each +column, but any appending or concatenating will require copying. Making a :class:`RecordBatch` +has two steps, given existing :class:`Arrays <Array>`: + +#. Defining a :class:`Schema` +#. Loading the :class:`Schema` and Arrays into the constructor + +Defining a Schema +^^^^^^^^^^^^^^^^^ + +To get started making a :class:`RecordBatch`, we first need to define +characteristics of the columns, each represented by a :class:`Field` instance. +Each :class:`Field` contains a name and datatype for its associated column; then, +a :class:`Schema` groups them together and sets the order of the columns, like +so: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: Schema) + :end-before: (Doc section: Schema) + +Building a RecordBatch +^^^^^^^^^^^^^^^^^^^^^^ + +With data in :class:`Arrays <Array>` from the previous section, and column descriptions in our +:class:`Schema` from the previous step, we can make the :class:`RecordBatch`. Note that the +length of the columns is necessary, and the length is shared by all columns. + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: RBatch) + :end-before: (Doc section: RBatch) + +Now, we have our data in a nice tabular form, safely within the :class:`RecordBatch`. +What we can do with this will be discussed in the later tutorials. + +Making a ChunkedArray +--------------------- + +Let’s say that we want an array made up of sub-arrays, because it +can be useful for logic, for parallelizing work, for fitting each chunk +cutely into cache, or for exceeding the 2,147,483,647 row limit in a +standard Arrow :class:`Array`. For this, Arrow offers :class:`ChunkedArray`, which can be +made up of individual Arrow :class:`Arrays <Array>`. So, let’s build a few more :class:`Arrays <Array>`, Review Comment: I'm not sure what "logic" refers to (though leave in if there's a good reason), but seems like we are missing the most immediate reason, which is avoiding data copies when extending / concatenating. ```suggestion can be useful for avoiding data copies when concatenating, for parallelizing work, for fitting each chunk cutely into cache, or for exceeding the 2,147,483,647 row limit in a standard Arrow :class:`Array`. For this, Arrow offers :class:`ChunkedArray`, which can be made up of individual Arrow :class:`Arrays <Array>`. In this example, we can reuse the arrays we made earlier in part of our chunked array, allowing us to extend them without having to copy data. So, let’s build a few more :class:`Arrays <Array>`, ``` ########## docs/source/cpp/tutorials/io_tutorial.rst: ########## @@ -0,0 +1,404 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +============== +Arrow File I/O +============== + +Apache Arrow provides file I/O functions to facilitate use of Arrow from +the start to end of an application. In this article, you will: + +1. Read an Arrow file into a :class:`RecordBatch` and write it back out afterwards + +2. Read a CSV file into a :class:`Table` and write it back out afterwards + +3. Read a Parquet file into a :class:`Table` and write it back out afterwards + +Pre-requisites +--------------- + +Before continuing, make sure you have: + +1. An Arrow installation + +2. An understanding of basic Arrow data structures from <the preceding article> + +3. A directory to run the final application in – this program will generate some files, so be prepared for that. + +Setup +----- + +Before writing out some file I/O, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. A ``main()`` is needed to glue things together. + +3. We need files to play with. + +Includes +^^^^^^^^ + +Before writing C++ code, we need some includes. We'll get ``iostream`` for output, then import Arrow's +I/O functionality for each file type we'll work with in this article: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +For our glue, we’ll use the ``main()`` pattern from the previous tutorial on +data structures: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +Which, like when we used it before, is paired with a ``RunMain()``: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: RunMain) + :end-before: (Doc section: RunMain) + +Generating Files for Reading +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We need some files to actually play with. In practice, you’ll likely +have some input for your own application. Here, however, we want to +explore doing I/O for the sake of it, so let’s generate some files to make +this easy to follow. To create those, we’ll define a helper function +that we’ll run first. Feel free to read through this, but the concepts +used will be explained later in this article. Note that we’re using the +day/month/year data from the previous tutorial. For now, just copy the +function in: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: GenInitialFile) + :end-before: (Doc section: GenInitialFile) + +To get the files for the rest of your code to function, make sure to +call ``GenInitialFile()`` as the very first line in ``RunMain()`` to initialize +the environment: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Gen Files) + :end-before: (Doc section: Gen Files) + +I/O with Arrow Files +-------------------- + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Bind file to :class:`ipc::RecordBatchFileReader` + + c. Read file to :class:`RecordBatch` + +2. Writing a file + + a. Get a :class:`io::FileOutputStream` + + b. Write to file from :class:`RecordBatch` + +Opening a File +^^^^^^^^^^^^^^ + +To actually read a file, we need to get some sort of way to point to it. +In Arrow, that means we’re going to get a :class:`io::ReadableFile` object – much +like an :class:`ArrayBuilder` can clear and make new arrays, we can reassign this +to new files, so we’ll use this instance throughout the examples: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: ReadableFile Definition) + :end-before: (Doc section: ReadableFile Definition) + +A :class:`io::ReadableFile` does little alone – we actually have it bind to a file +with :func:`io::ReadableFile::Open`. You can read more on that and its arguments at <link>. For +our purposes here, the default arguments suffice: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow ReadableFile Open) + :end-before: (Doc section: Arrow ReadableFile Open) + +Opening an Arrow file Reader +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An :class:`io::ReadableFile` is too generic to offer all functionality to read an Arrow file. +We need to use it to get an :class:`ipc::RecordBatchFileReader` object. This object implements +all the logic needed to read an Arrow file with correct formatting. We get one through +:func:`ipc::RecordBatchFileReader::Open`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read Open) + :end-before: (Doc section: Arrow Read Open) + +Reading an Open Arrow File to RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We have to use a :class:`RecordBatch` to read an Arrow file, so we’ll get a +:class:`RecordBatch`. Once we have that, we can actually read the file. Arrow +files can have multiple :class:`RecordBatches <RecordBatch>`, so we must pass an index. This +file only has one, so pass 0: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read) + :end-before: (Doc section: Arrow Read) + +Prepare a FileOutputStream +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For output, we need a :class:`io::FileOutputStream`. Just like our :class:`io::ReadableFile`, +we’ll be reusing this, so be ready for that. We open files the same way +as when reading: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Write Open) + :end-before: (Doc section: Arrow Write Open) + +Write Arrow File from RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, we grab our :class:`RecordBatch` we read into previously, and use it, along +with our target file, to create a :class:`ipc::RecordBatchWriter`. The +:class:`ipc::RecordBatchWriter` needs two things: + +1. the target file + +2. the :class:`Schema` for our :class:`RecordBatch` (in case we need to write more :class:`RecordBatches <RecordBatch>` of the same format.) + +The :class:`Schema` comes from our existing :class:`RecordBatch`, and the target file is +just a name – in this case, test_out.arrow. + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Writer) + :end-before: (Doc section: Arrow Writer) + +We can just call :func:`ipc::RecordBatchWriter::WriteRecordBatch` with our :class:`RecordBatch` to fill up our +file: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Write) + :end-before: (Doc section: Arrow Write) + +For IPC in particular, the writer has to be closed, so do that: Review Comment: ```suggestion For IPC in particular, the writer has to be closed since it anticipates more than one batch may be written. To do that: ``` ########## docs/source/cpp/tutorials/basic_arrow.rst: ########## @@ -0,0 +1,283 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +=========================== +Basic Arrow Data Structures +=========================== + +Apache Arrow provides fundamental data structures for representing data: +:class:`Array`, :class:`ChunkedArray`, :class:`RecordBatch`, and :class:`Table`. +This article shows how to construct these data structures from primitive +data types; specifically, we will work with integers of varying size +representing days, months, and years. We will use them to create the following data structures: + +#. Arrow :class:`Arrays <Array>` +#. :class:`ChunkedArrays<ChunkedArray>` +#. :class:`RecordBatch`, from :class:`Arrays <Array>` +#. :class:`Table`, from :class:`ChunkedArrays<ChunkedArray>` + +Pre-requisites +-------------- +Before continuing, make sure you have: + +#. An Arrow installation +#. Understanding of how to use basic C++ data structures +#. Understanding of basic C++ data types + + +Setup +----- + +Before trying out Arrow, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. ``A main()`` is needed to glue things together. + +Includes +^^^^^^^^ + +First, as ever, we need some includes. We'll get ``iostream`` for output, then import Arrow's basic +functionality from ``api.h``, like so: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +Next, we need a ``main()`` – a common pattern with Arrow looks like the +following: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +This allows us to easily use Arrow’s error-handling macros, which will +return back to ``main()`` with a :class:`arrow::Status` object if a failure occurs – and +this ``main()`` will report the error. Note that this means Arrow never +raises exceptions, instead relying upon returning :class:`Status`. For more on +that, read here: <LINK TO CONCEPTUAL OVERVIEW> + +To accompany this ``main()``, we have a ``RunMain()`` from which any :class:`Status` +objects can return – this is where we’ll write the rest of the program: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/arrow_example.cc + :language: cpp + :start-after: (Doc section: RunMain Start) + :end-before: (Doc section: RunMain Start) + + +Making an Arrow Array +--------------------- + +Building int8 Arrays +^^^^^^^^^^^^^^^^^^^^ + +Given that we have some data in standard C++ arrays, and want to use Arrow, we need to move +the data from said arrays into Arrow arrays. We still guarantee contiguity of memory in an +:class:`Array`, so no worries about a performance loss when using :class:`Array` vs C++ arrays. +The easiest way to construct an :class:`Array` uses an :class:`ArrayBuilder`. <RST +NOTE NEAR HERE: for more technical details, check out…> Review Comment: This might be best as a cookbook topic, but it occurred to me while reading this again that we should describe the various ways you can iterate over values in an Array (including validity) and discuss the performance and convenience trade-offs. ########## cpp/examples/tutorial_examples/dataset_example.cc: ########## @@ -0,0 +1,235 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +// (Doc section: Dataset Example) + +// (Doc section: Includes) +#include <arrow/api.h> +#include <arrow/dataset/api.h> +#include <parquet/arrow/reader.h> +#include <parquet/arrow/writer.h> Review Comment: We should mention in a comment that the parquet includes are only used for setup, and not necessary for using Datasets. ```suggestion // We use Parquet headers for setting up examples; they are not required for using datasets. #include <parquet/arrow/reader.h> #include <parquet/arrow/writer.h> ``` ########## docs/source/cpp/tutorials/io_tutorial.rst: ########## @@ -0,0 +1,404 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +============== +Arrow File I/O +============== + +Apache Arrow provides file I/O functions to facilitate use of Arrow from +the start to end of an application. In this article, you will: + +1. Read an Arrow file into a :class:`RecordBatch` and write it back out afterwards + +2. Read a CSV file into a :class:`Table` and write it back out afterwards + +3. Read a Parquet file into a :class:`Table` and write it back out afterwards + +Pre-requisites +--------------- + +Before continuing, make sure you have: + +1. An Arrow installation + +2. An understanding of basic Arrow data structures from <the preceding article> + +3. A directory to run the final application in – this program will generate some files, so be prepared for that. + +Setup +----- + +Before writing out some file I/O, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. A ``main()`` is needed to glue things together. + +3. We need files to play with. + +Includes +^^^^^^^^ + +Before writing C++ code, we need some includes. We'll get ``iostream`` for output, then import Arrow's +I/O functionality for each file type we'll work with in this article: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +For our glue, we’ll use the ``main()`` pattern from the previous tutorial on +data structures: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +Which, like when we used it before, is paired with a ``RunMain()``: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: RunMain) + :end-before: (Doc section: RunMain) + +Generating Files for Reading +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We need some files to actually play with. In practice, you’ll likely +have some input for your own application. Here, however, we want to +explore doing I/O for the sake of it, so let’s generate some files to make +this easy to follow. To create those, we’ll define a helper function +that we’ll run first. Feel free to read through this, but the concepts +used will be explained later in this article. Note that we’re using the +day/month/year data from the previous tutorial. For now, just copy the +function in: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: GenInitialFile) + :end-before: (Doc section: GenInitialFile) + +To get the files for the rest of your code to function, make sure to +call ``GenInitialFile()`` as the very first line in ``RunMain()`` to initialize +the environment: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Gen Files) + :end-before: (Doc section: Gen Files) + +I/O with Arrow Files +-------------------- + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Bind file to :class:`ipc::RecordBatchFileReader` + + c. Read file to :class:`RecordBatch` + +2. Writing a file + + a. Get a :class:`io::FileOutputStream` + + b. Write to file from :class:`RecordBatch` + +Opening a File +^^^^^^^^^^^^^^ + +To actually read a file, we need to get some sort of way to point to it. +In Arrow, that means we’re going to get a :class:`io::ReadableFile` object – much +like an :class:`ArrayBuilder` can clear and make new arrays, we can reassign this +to new files, so we’ll use this instance throughout the examples: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: ReadableFile Definition) + :end-before: (Doc section: ReadableFile Definition) + +A :class:`io::ReadableFile` does little alone – we actually have it bind to a file +with :func:`io::ReadableFile::Open`. You can read more on that and its arguments at <link>. For +our purposes here, the default arguments suffice: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow ReadableFile Open) + :end-before: (Doc section: Arrow ReadableFile Open) + +Opening an Arrow file Reader +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An :class:`io::ReadableFile` is too generic to offer all functionality to read an Arrow file. +We need to use it to get an :class:`ipc::RecordBatchFileReader` object. This object implements +all the logic needed to read an Arrow file with correct formatting. We get one through +:func:`ipc::RecordBatchFileReader::Open`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read Open) + :end-before: (Doc section: Arrow Read Open) + +Reading an Open Arrow File to RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We have to use a :class:`RecordBatch` to read an Arrow file, so we’ll get a +:class:`RecordBatch`. Once we have that, we can actually read the file. Arrow +files can have multiple :class:`RecordBatches <RecordBatch>`, so we must pass an index. This +file only has one, so pass 0: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Read) + :end-before: (Doc section: Arrow Read) + +Prepare a FileOutputStream +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For output, we need a :class:`io::FileOutputStream`. Just like our :class:`io::ReadableFile`, +we’ll be reusing this, so be ready for that. We open files the same way +as when reading: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Write Open) + :end-before: (Doc section: Arrow Write Open) + +Write Arrow File from RecordBatch +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Now, we grab our :class:`RecordBatch` we read into previously, and use it, along +with our target file, to create a :class:`ipc::RecordBatchWriter`. The +:class:`ipc::RecordBatchWriter` needs two things: + +1. the target file + +2. the :class:`Schema` for our :class:`RecordBatch` (in case we need to write more :class:`RecordBatches <RecordBatch>` of the same format.) + +The :class:`Schema` comes from our existing :class:`RecordBatch`, and the target file is +just a name – in this case, test_out.arrow. + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Writer) + :end-before: (Doc section: Arrow Writer) + +We can just call :func:`ipc::RecordBatchWriter::WriteRecordBatch` with our :class:`RecordBatch` to fill up our +file: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Write) + :end-before: (Doc section: Arrow Write) + +For IPC in particular, the writer has to be closed, so do that: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow Close) + :end-before: (Doc section: Arrow Close) + +Now we’ve read and written an IPC file! + +I/O with CSV +------------ + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Prepare Table + + c. Read File using :class:`csv::TableReader` + +2. Writing a file + + a. Get a :class:`io::FileOutputStream` + + b. Write to file from :class:`Table` + +Opening a CSV File +^^^^^^^^^^^^^^^^^^ + +For a CSV file, we need to open a :class:`io::ReadableFile`, just like an Arrow file, +and reuse our :class:`io::ReadableFile` object from before to do so: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Read Open) + :end-before: (Doc section: CSV Read Open) + +Preparing a Table +^^^^^^^^^^^^^^^^^ + +CSV can be read into a :class:`Table`, so declare a pointer to a :class:`Table`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Table Declare) + :end-before: (Doc section: CSV Table Declare) + +Read a CSV File to Table +^^^^^^^^^^^^^^^^^^^^^^^^ + +The CSV reader has option structs which need to be passed – luckily, +there are defaults for these which we can pass directly. For reference +on the other options, go here: <link>. This CSV is a standard CSV +without any special delimiters and is small, so we can make our reader +with defaults: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Reader Make) + :end-before: (Doc section: CSV Reader Make) + +With the CSV reader primed, we can use its :func:`csv::TableReader::Read` method to fill our +:class:`Table`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Read) + :end-before: (Doc section: CSV Read) + +Write a CSV File from Table +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +CSV writing to :class:`Table` looks exactly like IPC writing to :class:`RecordBatch`, +except with our :class:`Table`, and using :func:`ipc::RecordBatchWriter::WriteTable` instead of +:func:`ipc::RecordBatchWriter::WriteRecordBatch`. Note that the same writer class is used -- +we're writing with :func:`ipc::RecordBatchWriter::WriteTable` because we have a :class:`Table`. We’ll target +a file, use our :class:`Table’s <Table>` :class:`Schema`, and then write the :class:`Table`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: CSV Write) + :end-before: (Doc section: CSV Write) + +Now, we’ve read and written a CSV file! + +File I/O with Parquet +--------------------- + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Prepare :class:`parquet::arrow::FileReader` + + c. Read file to :class:`Table` + +2. Writing a file + + a. Write :class:`Table` to file + +Opening a Parquet File +^^^^^^^^^^^^^^^^^^^^^^ + +Once more, this file format, Parquet, needs a :class:`io::ReadableFile`, which we +already have, and for the :func:`io::ReadableFile::Open` method to be called on a file: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Parquet Read Open) + :end-before: (Doc section: Parquet Read Open) + +Setting up a Parquet Reader +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As always, we need a Reader to actually read the file. We’ve been +getting Readers for each file format from the Arrow namespace. This +time, we enter the Parquet namespace to get the :class:`parquet::arrow::FileReader`: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Parquet FileReader) + :end-before: (Doc section: Parquet FileReader) + +Now, to set up our reader, we call :func:`parquet::arrow::OpenFile`. Yes, this is necessary +even though we used :func:`io::ReadableFile::Open`. Note that we pass our +:class:`parquet::arrow::FileReader` by reference, instead of assigning to it in output: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Parquet OpenFile) + :end-before: (Doc section: Parquet OpenFile) + +Reading a Parquet File to Table +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +With a prepared :class:`parquet::arrow::FileReader` in hand, we can read to a +:class:`Table`, except we must pass the :class:`Table` by reference instead of outputting to it: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Parquet Read) + :end-before: (Doc section: Parquet Read) + +Writing a Parquet File from Table +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Writing a Parquet file does not need a writer object. Instead, we give Review Comment: ```suggestion For single-shot writes, writing a Parquet file does not need a writer object. Instead, we give ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
