Copilot commented on code in PR #50419: URL: https://github.com/apache/arrow/pull/50419#discussion_r3551230211
########## docs/source/python/parquet/parquet_datasets.rst: ########## @@ -0,0 +1,251 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_datasets: + + +Partitioned Datasets (Multiple Files) +===================================== + +Multiple Parquet files constitute a Parquet *dataset*. These may present in a +number of ways: Review Comment: Grammar: "These may present" is missing "be" (should be "may be present"). ########## docs/source/python/parquet/parquet_type_handling.rst: ########## @@ -0,0 +1,156 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_type_handling: + + +Data Type Handling +================== + +Reading types as DictionaryArray +-------------------------------- + +The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will +cause columns to be read as ``DictionaryArray``, which will become +``pandas.Categorical`` when converted to pandas. This option is only valid for +string and binary column types, and it can yield significantly lower memory use +and improved performance for columns with many repeated string values. + +.. code-block:: python + + >>> import pyarrow as pa + >>> import pyarrow.parquet as pq + + >>> table = pa.table({'one': [-1, None, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}) + ... + >>> pq.write_table(table, 'example.parquet') + + >>> pq.read_table('example.parquet', read_dictionary=['two']) + pyarrow.Table + one: double + two: dictionary<values=string, indices=int32, ordered=0> + three: bool + ---- + one: [[-1,null,2.5]] + two: [ -- dictionary: + ["foo","bar","baz"] -- indices: + [0,1,2]] + three: [[true,false,true]] + +Reading binary and list columns +------------------------------- + +By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type +and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use +32-bit offsets. For very large datasets this can overflow. The ``binary_type`` +and ``list_type`` parameters let you choose a different Arrow type on read. + +``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or +``pa.binary_view()``: + +.. code-block:: python + + >>> import pyarrow as pa + >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())}) + >>> pq.write_table(table, 'binary.parquet', store_schema=False) + >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema + data: large_binary + +``list_type`` accepts ``pa.ListType`` or ``pa.LargeListType``: + +.. code-block:: python + + >>> table = pa.table({'lists': pa.array([[1, 2], [3]], pa.list_(pa.int32()))}) + >>> pq.write_table(table, 'lists.parquet', store_schema=False) + >>> pq.read_table('lists.parquet', list_type=pa.LargeListType).schema + lists: large_list<element: int32> + child 0, element: int32 + +.. note:: + Both settings are ignored when a serialized Arrow schema is present in the + Parquet file metadata (i.e. when the parquet file was written with + ``store_schema=True``). + +Read in Arrow Extension Types +----------------------------- + +Certain Parquet logical types (JSON, UUID, Geometry, Geography) are supported +and read as Arrow extension types by default (``arrow.json``, ``arrow.uuid`` +and ``geoarrow.wkb`` respectively). This support is enabled via the +``arrow_extensions_enabled`` parameter and is used in :func:`~pyarrow.parquet.read_table`, +:class:`~pyarrow.parquet.ParquetFile`, :class:`~pyarrow.parquet.ParquetDataset`, and +:func:`~pyarrow.parquet.read_schema`. + +To read these Parquet logical types as storage types (default behavior +until PyArrow version ``21.0.0``), set ``arrow_extensions_enabled=False``. + +.. note:: + Reading GEOMETRY/GEOGRAPHY columns as ``geoarrow.wkb`` additionally + requires the ``geoarrow.wkb`` extension type to be registered. For that + you can install Python bindings for GeoArrow + (`geoarrow-pyarrow <https://geoarrow.org/geoarrow-python/main/index.html#installation>`_) + and import ``geoarrow.pyarrow`` module. + +Storing timestamps +------------------ + +Some Parquet readers may only support timestamps stored in millisecond +(``'ms'``) or microsecond (``'us'``) resolution. Since pandas uses nanoseconds +to represent timestamps, this can occasionally be a nuisance. By default +(when writing version 1.0 Parquet files), the nanoseconds will be cast to +microseconds ('us'). + +In addition, We provide the ``coerce_timestamps`` option to allow you to select Review Comment: Mid-sentence capitalization: "We" should be lowercase here. ########## docs/source/python/parquet/parquet_encryption.rst: ########## @@ -0,0 +1,280 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +------------------------------------------------ + +Columnar encryption is supported for Parquet files in C++ starting from +Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. + +Parquet uses the envelope encryption practice, where file parts are encrypted +with "data encryption keys" (DEKs), and the DEKs are encrypted with "master +encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each +encrypted file/column. The MEKs are generated, stored and managed in a Key +Management Service (KMS) of user’s choice. + +Reading and writing encrypted Parquet files involves passing file encryption +and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to +:class:`~.ParquetFile`, respectively. + +Writing an encrypted Parquet file: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP + ... kms_connection_config, encryption_config) + >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP + ... encryption_properties=encryption_properties) as writer: + ... writer.write_table(table) + +Reading an encrypted Parquet file: + +.. code-block:: python + + >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP + ... kms_connection_config) + >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP + ... decryption_properties=decryption_properties) + + +In order to create the encryption and decryption properties, a +:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and +initialized with KMS Client details, as described below. + + +KMS Client +~~~~~~~~~~ + +The master encryption keys should be kept and managed in a production-grade +Key Management System (KMS), deployed in the user's organization. Using Parquet +encryption requires implementation of a client class for the KMS server. +Any KmsClient implementation should implement the informal interface +defined by :class:`pyarrow.parquet.encryption.KmsClient` as following: + +.. code-block:: python + + >>> import pyarrow.parquet.encryption as pe + >>> class MyKmsClient(pe.KmsClient): + ... + ... """An example KmsClient implementation skeleton""" + ... def __init__(self, kms_connection_configuration): + ... pe.KmsClient.__init__(self) + ... # Any KMS-specific initialization based on + ... # kms_connection_configuration comes here + ... + ... def wrap_key(self, key_bytes, master_key_identifier): + ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by + ... # master_key_identifier + ... return wrapped_key + ... + ... def unwrap_key(self, wrapped_key, master_key_identifier): + ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by + ... # master_key_identifier + ... return key_bytes + +The concrete implementation will be loaded at runtime by a factory function +provided by the user. This factory function will be used to initialize the +:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption +and decryption properties. + +For example, in order to use the ``MyKmsClient`` defined above: + +.. code-block:: python + + >>> def kms_client_factory(kms_connection_configuration): + ... return MyKmsClient(kms_connection_configuration) + + >>> crypto_factory = pe.CryptoFactory(kms_client_factory) + +An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>` Review Comment: The :download: link path was correct when this page lived at docs/source/python/parquet.rst, but after moving into docs/source/python/parquet/ it now points to a non-existent location. This will break the downloadable example link (and may error at doc build time). ########## docs/source/python/parquet/parquet_encryption.rst: ########## @@ -0,0 +1,280 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +------------------------------------------------ + +Columnar encryption is supported for Parquet files in C++ starting from +Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. + +Parquet uses the envelope encryption practice, where file parts are encrypted +with "data encryption keys" (DEKs), and the DEKs are encrypted with "master +encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each +encrypted file/column. The MEKs are generated, stored and managed in a Key +Management Service (KMS) of user’s choice. + +Reading and writing encrypted Parquet files involves passing file encryption +and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to +:class:`~.ParquetFile`, respectively. + +Writing an encrypted Parquet file: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP + ... kms_connection_config, encryption_config) + >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP + ... encryption_properties=encryption_properties) as writer: + ... writer.write_table(table) + +Reading an encrypted Parquet file: + +.. code-block:: python + + >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP + ... kms_connection_config) + >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP + ... decryption_properties=decryption_properties) + + +In order to create the encryption and decryption properties, a +:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and +initialized with KMS Client details, as described below. + + +KMS Client +~~~~~~~~~~ + +The master encryption keys should be kept and managed in a production-grade +Key Management System (KMS), deployed in the user's organization. Using Parquet +encryption requires implementation of a client class for the KMS server. +Any KmsClient implementation should implement the informal interface +defined by :class:`pyarrow.parquet.encryption.KmsClient` as following: Review Comment: Grammar: "as following" should be "as follows". ########## docs/source/python/index.rst: ########## @@ -52,17 +52,17 @@ files into Arrow structures. filesystems numpy pandas - interchange_protocol - dlpack timestamps - orc + parquet/index + dataset Review Comment: The PR description says "No user-facing changes", but switching the top-level Parquet page from python/parquet.rst to python/parquet/index.rst typically changes the built URL (Sphinx defaults to directory-style URLs unless configured otherwise) and can break existing inbound links/bookmarks. Consider preserving the old doc name/URL (e.g., avoid introducing a parquet/ subdir), or add an explicit redirect from the old location to the new one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
