Copilot commented on code in PR #50419: URL: https://github.com/apache/arrow/pull/50419#discussion_r3615286194
########## docs/source/python/parquet/parquet_encryption.rst: ########## @@ -0,0 +1,280 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +================================================ + +Columnar encryption is supported for Parquet files in C++ starting from +Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. + +Parquet uses the envelope encryption practice, where file parts are encrypted +with "data encryption keys" (DEKs), and the DEKs are encrypted with "master +encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each +encrypted file/column. The MEKs are generated, stored and managed in a Key +Management Service (KMS) of user’s choice. + +Reading and writing encrypted Parquet files involves passing file encryption +and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to +:class:`~.ParquetFile`, respectively. Review Comment: `:class:`~.ParquetFile`` resolves to `pyarrow.ParquetFile` because of `.. currentmodule:: pyarrow`, but `ParquetFile` lives in `pyarrow.parquet`. This will produce a broken API link in the rendered docs; reference the fully-qualified class instead. ########## docs/source/python/parquet/parquet_datasets.rst: ########## @@ -0,0 +1,251 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_datasets: + + +Partitioned Datasets (Multiple Files) +===================================== + +Multiple Parquet files constitute a Parquet *dataset*. These may be present in a +number of ways: + +* A list of Parquet absolute file paths +* A directory name containing nested directories defining a partitioned dataset + +A dataset partitioned by year and month may look like on disk: + +.. code-block:: text + + dataset_name/ + year=2007/ + month=01/ + 0.parq + 1.parq + ... + month=02/ + 0.parq + 1.parq + ... + month=03/ + ... + year=2008/ + month=01/ + ... + ... + +Writing to Partitioned Datasets +------------------------------- + +You can write a partitioned dataset for any ``pyarrow`` file system that is a +file-store (e.g. local, HDFS, S3). The default behaviour when no filesystem is +added is to use the local filesystem. + +.. code-block:: python + + >>> import pyarrow as pa + >>> import pyarrow.parquet as pq + + >>> table = pa.table({'one': [-1, None, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}) + ... + + >>> # Local dataset write + >>> pq.write_to_dataset(table, root_path='dataset_name', + ... partition_cols=['one', 'two']) + +The root path in this case specifies the parent directory to which data will be +saved. The partition columns are the column names by which to partition the +dataset. Columns are partitioned in the order they are given. The partition +splits are determined by the unique values in the partition columns. + +To use another filesystem you only need to add the filesystem parameter, the +individual table writes are wrapped using ``with`` statements so the +``pq.write_to_dataset`` function does not need to be. + +.. code-block:: python + + >>> # Remote file-system example + >>> from pyarrow.fs import HadoopFileSystem # doctest: +SKIP + >>> fs = HadoopFileSystem(host, port, user=user, kerb_ticket=ticket_cache_path) # doctest: +SKIP + >>> pq.write_to_dataset(table, root_path='dataset_name', # doctest: +SKIP + ... partition_cols=['one', 'two'], filesystem=fs) + +Compatibility Note: if using ``pq.write_to_dataset`` to create a table that +will then be used by HIVE then partition column values must be compatible with +the allowed character set of the HIVE version you are running. + +Writing ``_metadata`` and ``_common_metadata`` files +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some processing frameworks such as Spark or Dask (optionally) use ``_metadata`` +and ``_common_metadata`` files with partitioned datasets. + +Those files include information about the schema of the full dataset (for +``_common_metadata``) and potentially all row group metadata of all files in the +partitioned dataset as well (for ``_metadata``). The actual files are +metadata-only Parquet files. Note this is not a Parquet standard, but a +convention set in practice by those frameworks. + +Using those files can give a more efficient creation of a parquet Dataset, +since it can use the stored schema and file paths of all row groups, +instead of inferring the schema and crawling the directories for all Parquet +files (this is especially the case for filesystems where accessing files +is expensive). + +The :func:`~pyarrow.parquet.write_to_dataset` function does not automatically +write such metadata files, but you can use it to gather the metadata and +combine and write them manually: + +.. code-block:: python + + >>> # Write a dataset and collect metadata information of all written files + >>> metadata_collector = [] + >>> root_path = "dataset_name_1" + >>> pq.write_to_dataset(table, root_path, metadata_collector=metadata_collector) + + >>> # Write the ``_common_metadata`` parquet file without row groups statistics + >>> pq.write_metadata(table.schema, root_path + '/_common_metadata') + + >>> # Write the ``_metadata`` parquet file with row groups statistics of all files + >>> pq.write_metadata( + ... table.schema, root_path + '/_metadata', + ... metadata_collector=metadata_collector + ... ) + +When not using the :func:`~pyarrow.parquet.write_to_dataset` function, but +writing the individual files of the partitioned dataset using +:func:`~pyarrow.parquet.write_table` or :class:`~pyarrow.parquet.ParquetWriter`, +the ``metadata_collector`` keyword can also be used to collect the FileMetaData +of the written files. In this case, you need to ensure to set the file path +contained in the row group metadata yourself before combining the metadata, and +the schemas of all different files and collected FileMetaData objects should be +the same: + +.. code-block:: python + + >>> import os + >>> os.mkdir("year=2017") + + >>> metadata_collector = [] + >>> pq.write_table( + ... table, "year=2017/data1.parquet", + ... metadata_collector=metadata_collector + ... ) + + >>> # set the file path relative to the root of the partitioned dataset + >>> metadata_collector[-1].set_file_path("year=2017/data1.parquet") + + >>> # combine and write the metadata + >>> metadata = metadata_collector[0] + >>> for _meta in metadata_collector[1:]: + ... metadata.append_row_groups(_meta) + >>> metadata.write_metadata_file("_metadata") + + >>> # or use pq.write_metadata to combine and write in a single step + >>> pq.write_metadata( + ... table.schema, "_metadata", + ... metadata_collector=metadata_collector + ... ) + + >>> pq.read_metadata("_metadata") + <pyarrow._parquet.FileMetaData object at ...> + created_by: parquet-cpp-arrow version ... + num_columns: 3 + num_rows: 3 + num_row_groups: 1 + format_version: 2.6 + serialized_size: ... + +Reading from Partitioned Datasets +--------------------------------- + +The :class:`~.ParquetDataset` class accepts either a directory name or a list Review Comment: `:class:`~.ParquetDataset`` resolves to `pyarrow.ParquetDataset` under `.. currentmodule:: pyarrow`, but `ParquetDataset` is in `pyarrow.parquet`. This will render as an unresolved/broken cross-reference. ########## docs/source/python/parquet/index.rst: ########## @@ -0,0 +1,89 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet: + +Reading and Writing the Apache Parquet Format +============================================= + +The `Apache Parquet <http://parquet.apache.org/>`_ project provides a +standardized open-source columnar storage format for use in data analysis +systems. It was created originally for use in `Apache Hadoop +<http://hadoop.apache.org/>`_ with systems like `Apache Drill +<http://drill.apache.org>`_, `Apache Hive <http://hive.apache.org>`_, `Apache +Impala <http://impala.apache.org>`_, and `Apache Spark +<http://spark.apache.org>`_ adopting it as a shared standard for high +performance data IO. + +Apache Arrow is an ideal in-memory transport layer for data that is being read +or written with Parquet files. We have been concurrently developing the `C++ +implementation of +Apache Parquet <https://github.com/apache/arrow/tree/main/cpp/tools/parquet>`_, +which includes a native, multithreaded C++ adapter to and from in-memory Arrow +data. PyArrow includes Python bindings to this code, which thus enables reading +and writing Parquet files with pandas as well. + +Obtaining pyarrow with Parquet Support +-------------------------------------- + +If you installed ``pyarrow`` with pip or conda, it should be built with Parquet +support bundled: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + +If you are building ``pyarrow`` from source, you must use ``-DARROW_PARQUET=ON`` +when compiling the C++ libraries and enable the Parquet extensions when +building ``pyarrow``. If you want to use Parquet Encryption, then you must +use ``-DPARQUET_REQUIRE_ENCRYPTION=ON`` too when compiling the C++ libraries. +See the :ref:`Python Development <python-development>` page for more details. + + +Reading and Writing Single Files +-------------------------------- + +.. toctree:: + :maxdepth: 2 + + parquet + +Data Type Handling +------------------ + +.. toctree:: + :maxdepth: 2 + + parquet_type_handling + Review Comment: PR description says the Parquet docs are split into 4 subpages, but this toctree adds an additional dedicated page for type handling (`parquet_type_handling`). Please update the PR description (or adjust the split) so the described scope matches what will be published. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
