Copilot commented on code in PR #50419: URL: https://github.com/apache/arrow/pull/50419#discussion_r3615192219
########## docs/source/python/parquet/parquet.rst: ########## @@ -0,0 +1,605 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_single_file: + + +Reading and Writing Single Files +================================ + +The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table` +read and write the :ref:`pyarrow.Table <data.table>` object, respectively. + +Let's look at a simple table: + +.. code-block:: python + + >>> import numpy as np + >>> import pandas as pd + >>> import pyarrow as pa + >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}, + ... index=list('abc')) + >>> table = pa.Table.from_pandas(df) + +We write this to Parquet format with ``write_table``: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> pq.write_table(table, 'example.parquet') + +This creates a single Parquet file. In practice, a Parquet dataset may consist +of many files in many directories. We can read a single file back with +``read_table``: + +.. code-block:: python + + >>> table2 = pq.read_table('example.parquet') + >>> table2.to_pandas() + one two three + a -1.0 foo True + b NaN bar False + c 2.5 baz True + +You can pass a subset of columns to read, which can be much faster than reading +the whole file (due to the columnar layout): + +.. code-block:: python + + >>> pq.read_table('example.parquet', columns=['one', 'three']) + pyarrow.Table + one: double + three: bool + ---- + one: [[-1,null,2.5]] + three: [[true,false,true]] + +When reading a subset of columns from a file that used a Pandas dataframe as the +source, we use ``read_pandas`` to maintain any additional index column data: + +.. code-block:: python + + >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas() + two + a foo + b bar + c baz + +We do not need to use a string to specify the origin of the file. It can be any of: + +* A file path as a string +* A :ref:`NativeFile <io.native_file>` from PyArrow +* A Python file object + +In general, a Python file object will have the worst read performance, while a +string file path or an instance of :class:`~.NativeFile` (especially memory +maps) will perform the best. + +.. _parquet_mmap: + +Reading Parquet and Memory Mapping +---------------------------------- + +Because Parquet data needs to be decoded from the Parquet format +and compression, it can't be directly mapped from disk. +Thus the ``memory_map`` option might perform better on some systems +but won't help much with resident memory consumption. + +.. code-block:: python + + >>> pq_array = pq.read_table(path, memory_map=True) # doctest: +SKIP + >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP + RSS: 4299MB + + >>> pq_array = pq.read_table(path, memory_map=False) # doctest: +SKIP + >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP + RSS: 4299MB + +If you need to deal with Parquet data bigger than memory, +the :ref:`dataset` and partitioning is probably what you are looking for. Review Comment: Grammar: the subject is plural ("dataset and partitioning"), so the verb should be plural as well. ########## docs/source/python/parquet/parquet_encryption.rst: ########## @@ -0,0 +1,280 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +------------------------------------------------ Review Comment: This page title is marked as a lower-level heading ("-") instead of the top-level document title adornment used elsewhere in the Python docs ("="). Using a top-level title improves consistency and avoids odd TOC nesting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
