ksuarez1423 commented on code in PR #13859: URL: https://github.com/apache/arrow/pull/13859#discussion_r972232312
########## docs/source/cpp/tutorials/io_tutorial.rst: ########## @@ -0,0 +1,404 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. default-domain:: cpp +.. highlight:: cpp + +.. cpp:namespace:: arrow + +============== +Arrow File I/O +============== + +Apache Arrow provides file I/O functions to facilitate use of Arrow from +the start to end of an application. In this article, you will: + +1. Read an Arrow file into a :class:`RecordBatch` and write it back out afterwards + +2. Read a CSV file into a :class:`Table` and write it back out afterwards + +3. Read a Parquet file into a :class:`Table` and write it back out afterwards + +Pre-requisites +--------------- + +Before continuing, make sure you have: + +1. An Arrow installation + +2. An understanding of basic Arrow data structures from :doc:`/cpp/tutorials/basic_arrow` + +3. A directory to run the final application in – this program will generate some files, so be prepared for that. + +Setup +----- + +Before writing out some file I/O, we need to fill in a couple gaps: + +1. We need to include necessary headers. + +2. A ``main()`` is needed to glue things together. + +3. We need files to play with. + +Includes +^^^^^^^^ + +Before writing C++ code, we need some includes. We'll get ``iostream`` for output, then import Arrow's +I/O functionality for each file type we'll work with in this article: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Includes) + :end-before: (Doc section: Includes) + +Main() +^^^^^^ + +For our glue, we’ll use the ``main()`` pattern from the previous tutorial on +data structures: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Main) + :end-before: (Doc section: Main) + +Which, like when we used it before, is paired with a ``RunMain()``: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: RunMain) + :end-before: (Doc section: RunMain) + +Generating Files for Reading +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We need some files to actually play with. In practice, you’ll likely +have some input for your own application. Here, however, we want to +explore doing I/O for the sake of it, so let’s generate some files to make +this easy to follow. To create those, we’ll define a helper function +that we’ll run first. Feel free to read through this, but the concepts +used will be explained later in this article. Note that we’re using the +day/month/year data from the previous tutorial. For now, just copy the +function in: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: GenInitialFile) + :end-before: (Doc section: GenInitialFile) + +To get the files for the rest of your code to function, make sure to +call ``GenInitialFile()`` as the very first line in ``RunMain()`` to initialize +the environment: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Gen Files) + :end-before: (Doc section: Gen Files) + +I/O with Arrow Files +-------------------- + +We’re going to go through this step by step, reading then writing, as +follows: + +1. Reading a file + + a. Open the file + + b. Bind file to :class:`ipc::RecordBatchFileReader` + + c. Read file to :class:`RecordBatch` + +2. Writing a file + + a. Get a :class:`io::FileOutputStream` + + b. Write to file from :class:`RecordBatch` + +Opening a File +^^^^^^^^^^^^^^ + +To actually read a file, we need to get some sort of way to point to it. +In Arrow, that means we’re going to get a :class:`io::ReadableFile` object – much +like an :class:`ArrayBuilder` can clear and make new arrays, we can reassign this +to new files, so we’ll use this instance throughout the examples: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: ReadableFile Definition) + :end-before: (Doc section: ReadableFile Definition) + +A :class:`io::ReadableFile` does little alone – we actually have it bind to a file +with :func:`io::ReadableFile::Open`. For +our purposes here, the default arguments suffice: + +.. literalinclude:: ../../../../cpp/examples/tutorial_examples/file_access_example.cc + :language: cpp + :start-after: (Doc section: Arrow ReadableFile Open) + :end-before: (Doc section: Arrow ReadableFile Open) + +Opening an Arrow file Reader Review Comment: I'm going with Arrow files for now, and will swap in Arrow IPC if it becomes solidified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
