Copilot commented on code in PR #50419: URL: https://github.com/apache/arrow/pull/50419#discussion_r3615036047
########## docs/source/python/parquet/parquet.rst: ########## @@ -0,0 +1,605 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_single_file: + + +Reading and Writing Single Files +================================ + +The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table` +read and write the :ref:`pyarrow.Table <data.table>` object, respectively. + +Let's look at a simple table: + +.. code-block:: python + + >>> import numpy as np + >>> import pandas as pd + >>> import pyarrow as pa + >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}, + ... index=list('abc')) + >>> table = pa.Table.from_pandas(df) + +We write this to Parquet format with ``write_table``: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> pq.write_table(table, 'example.parquet') + +This creates a single Parquet file. In practice, a Parquet dataset may consist +of many files in many directories. We can read a single file back with +``read_table``: + +.. code-block:: python + + >>> table2 = pq.read_table('example.parquet') + >>> table2.to_pandas() + one two three + a -1.0 foo True + b NaN bar False + c 2.5 baz True + +You can pass a subset of columns to read, which can be much faster than reading +the whole file (due to the columnar layout): + +.. code-block:: python + + >>> pq.read_table('example.parquet', columns=['one', 'three']) + pyarrow.Table + one: double + three: bool + ---- + one: [[-1,null,2.5]] + three: [[true,false,true]] + +When reading a subset of columns from a file that used a Pandas dataframe as the +source, we use ``read_pandas`` to maintain any additional index column data: + +.. code-block:: python + + >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas() + two + a foo + b bar + c baz + +We do not need to use a string to specify the origin of the file. It can be any of: + +* A file path as a string +* A :ref:`NativeFile <io.native_file>` from PyArrow +* A Python file object + +In general, a Python file object will have the worst read performance, while a +string file path or an instance of :class:`~.NativeFile` (especially memory +maps) will perform the best. + +.. _parquet_mmap: + +Reading Parquet and Memory Mapping +---------------------------------- + +Because Parquet data needs to be decoded from the Parquet format +and compression, it can't be directly mapped from disk. +Thus the ``memory_map`` option might perform better on some systems +but won't help much with resident memory consumption. + +.. code-block:: python + + >>> pq_array = pa.parquet.read_table(path, memory_map=True) # doctest: +SKIP + >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP + RSS: 4299MB + + >>> pq_array = pa.parquet.read_table(path, memory_map=False) # doctest: +SKIP + >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP + RSS: 4299MB + +If you need to deal with Parquet data bigger than memory, +the :ref:`dataset` and partitioning is probably what you are looking for. + +Parquet file writing options +---------------------------- + +:func:`~pyarrow.parquet.write_table()` has a number of options to +control various settings when writing a Parquet file. + +* ``version``, the Parquet format version to use. ``'1.0'`` ensures + compatibility with older readers, while ``'2.4'`` and greater values + enable more Parquet types and encodings. +* ``data_page_size``, to control the approximate size of encoded data + pages within a column chunk. This currently defaults to 1MB. +* ``max_rows_per_page``, to cap the number of rows per data page within + a column chunk (default 20000). Smaller values reduce memory usage + during reads at the cost of more page metadata. +* ``flavor``, to set compatibility options particular to a Parquet + consumer like ``'spark'`` for Apache Spark. +* ``store_decimal_as_integer``, to store decimals with precision 1–18 + as ``int32`` or ``int64`` instead of ``fixed_len_byte_array``. + This produces more compact files but may not be supported by all readers. +* ``write_time_adjusted_to_utc``, to mark ``TIME`` columns as + adjusted to UTC (``isAdjustedToUTC=True``). When ``False`` (the default), + the time is treated as local/unknown timezone. +* ``write_page_index``, to write statistics to the page index + instead of writing it to each data page header. + Note that PyArrow does not yet use the page index on the read side. +* ``write_page_checksum``, to write a page checksum. Use with + ``page_checksum_verification=True`` on read to detect data corruption. +* ``sorting_columns``, to record the sort order of the data in each + row group's metadata. The writer does not sort the data nor does it verify + that the data is sorted. Readers can use this metadata to optimize queries. + + Sort order is expressed as a sequence of :class:`~pyarrow.parquet.SortingColumn` + objects. + +See the :func:`~pyarrow.parquet.write_table()` docstring for more details. + +There are some additional data type handling-specific options +described below. + +Omitting the DataFrame index +---------------------------- + +When using ``pa.Table.from_pandas`` to convert to an Arrow table, by default +one or more special columns are added to keep track of the index (row +labels). Storing the index takes extra space, so if your index is not valuable, +you may choose to omit it by passing ``preserve_index=False`` + +.. code-block:: python + + >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}, + ... index=list('abc')) + >>> table = pa.Table.from_pandas(df, preserve_index=False) + +Then we have: + +.. code-block:: python + + >>> pq.write_table(table, 'example_noindex.parquet') + >>> t = pq.read_table('example_noindex.parquet') + >>> t.to_pandas() + one two three + 0 -1.0 foo True + 1 NaN bar False + 2 2.5 baz True + +Here you see the index did not survive the round trip. + +Finer-grained Reading and Writing +--------------------------------- + +``read_table`` uses the :class:`~.ParquetFile` class, which has other features: Review Comment: The cross-reference `:class:`~.ParquetFile`` resolves to `pyarrow.ParquetFile` (which is not exported) rather than `pyarrow.parquet.ParquetFile`. This likely produces an unresolved reference and is inconsistent with `parquet_type_handling.rst`, which uses fully-qualified `pyarrow.parquet.*` refs. Consider updating related refs in this page too (e.g. `:meth:`~.ParquetFile.iter_batches`` and `:func:`~parquet.read_metadata``) and similar occurrences in `parquet_datasets.rst` / `parquet_encryption.rst`. ########## docs/source/python/parquet/parquet.rst: ########## @@ -0,0 +1,605 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_single_file: + + +Reading and Writing Single Files +================================ + +The functions :func:`~.parquet.read_table` and :func:`~.parquet.write_table` +read and write the :ref:`pyarrow.Table <data.table>` object, respectively. + +Let's look at a simple table: + +.. code-block:: python + + >>> import numpy as np + >>> import pandas as pd + >>> import pyarrow as pa + >>> df = pd.DataFrame({'one': [-1, np.nan, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}, + ... index=list('abc')) + >>> table = pa.Table.from_pandas(df) + +We write this to Parquet format with ``write_table``: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> pq.write_table(table, 'example.parquet') + +This creates a single Parquet file. In practice, a Parquet dataset may consist +of many files in many directories. We can read a single file back with +``read_table``: + +.. code-block:: python + + >>> table2 = pq.read_table('example.parquet') + >>> table2.to_pandas() + one two three + a -1.0 foo True + b NaN bar False + c 2.5 baz True + +You can pass a subset of columns to read, which can be much faster than reading +the whole file (due to the columnar layout): + +.. code-block:: python + + >>> pq.read_table('example.parquet', columns=['one', 'three']) + pyarrow.Table + one: double + three: bool + ---- + one: [[-1,null,2.5]] + three: [[true,false,true]] + +When reading a subset of columns from a file that used a Pandas dataframe as the +source, we use ``read_pandas`` to maintain any additional index column data: + +.. code-block:: python + + >>> pq.read_pandas('example.parquet', columns=['two']).to_pandas() + two + a foo + b bar + c baz + +We do not need to use a string to specify the origin of the file. It can be any of: + +* A file path as a string +* A :ref:`NativeFile <io.native_file>` from PyArrow +* A Python file object + +In general, a Python file object will have the worst read performance, while a +string file path or an instance of :class:`~.NativeFile` (especially memory +maps) will perform the best. + +.. _parquet_mmap: + +Reading Parquet and Memory Mapping +---------------------------------- + +Because Parquet data needs to be decoded from the Parquet format +and compression, it can't be directly mapped from disk. +Thus the ``memory_map`` option might perform better on some systems +but won't help much with resident memory consumption. + +.. code-block:: python + + >>> pq_array = pa.parquet.read_table(path, memory_map=True) # doctest: +SKIP + >>> print("RSS: {}MB".format(pa.total_allocated_bytes() >> 20)) # doctest: +SKIP + RSS: 4299MB + + >>> pq_array = pa.parquet.read_table(path, memory_map=False) # doctest: +SKIP Review Comment: The memory-mapping example uses `pa.parquet.read_table`, but `ParquetFile`/`read_table` live under `pyarrow.parquet` (imported as `pq` above). As written, `pa.parquet` is not otherwise used in the docs and is likely incorrect/inconsistent for readers. ########## docs/source/python/parquet/parquet_encryption.rst: ########## @@ -0,0 +1,280 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_encryption: + + +Parquet Modular Encryption (Columnar Encryption) +------------------------------------------------ + +Columnar encryption is supported for Parquet files in C++ starting from +Apache Arrow 4.0.0 and in PyArrow starting from Apache Arrow 6.0.0. + +Parquet uses the envelope encryption practice, where file parts are encrypted +with "data encryption keys" (DEKs), and the DEKs are encrypted with "master +encryption keys" (MEKs). The DEKs are randomly generated by Parquet for each +encrypted file/column. The MEKs are generated, stored and managed in a Key +Management Service (KMS) of user’s choice. + +Reading and writing encrypted Parquet files involves passing file encryption +and decryption properties to :class:`~pyarrow.parquet.ParquetWriter` and to +:class:`~.ParquetFile`, respectively. + +Writing an encrypted Parquet file: + +.. code-block:: python + + >>> import pyarrow.parquet as pq + >>> encryption_properties = crypto_factory.file_encryption_properties( # doctest: +SKIP + ... kms_connection_config, encryption_config) + >>> with pq.ParquetWriter(filename, schema, # doctest: +SKIP + ... encryption_properties=encryption_properties) as writer: + ... writer.write_table(table) + +Reading an encrypted Parquet file: + +.. code-block:: python + + >>> decryption_properties = crypto_factory.file_decryption_properties( # doctest: +SKIP + ... kms_connection_config) + >>> parquet_file = pq.ParquetFile(filename, # doctest: +SKIP + ... decryption_properties=decryption_properties) + + +In order to create the encryption and decryption properties, a +:class:`pyarrow.parquet.encryption.CryptoFactory` should be created and +initialized with KMS Client details, as described below. + + +KMS Client +~~~~~~~~~~ + +The master encryption keys should be kept and managed in a production-grade +Key Management System (KMS), deployed in the user's organization. Using Parquet +encryption requires implementation of a client class for the KMS server. +Any KmsClient implementation should implement the informal interface +defined by :class:`pyarrow.parquet.encryption.KmsClient` as following: + +.. code-block:: python + + >>> import pyarrow.parquet.encryption as pe + >>> class MyKmsClient(pe.KmsClient): + ... + ... """An example KmsClient implementation skeleton""" + ... def __init__(self, kms_connection_configuration): + ... pe.KmsClient.__init__(self) + ... # Any KMS-specific initialization based on + ... # kms_connection_configuration comes here + ... + ... def wrap_key(self, key_bytes, master_key_identifier): + ... wrapped_key = ... # call KMS to wrap key_bytes with key specified by + ... # master_key_identifier + ... return wrapped_key + ... + ... def unwrap_key(self, wrapped_key, master_key_identifier): + ... key_bytes = ... # call KMS to unwrap wrapped_key with key specified by + ... # master_key_identifier + ... return key_bytes + +The concrete implementation will be loaded at runtime by a factory function +provided by the user. This factory function will be used to initialize the +:class:`pyarrow.parquet.encryption.CryptoFactory` for creating file encryption +and decryption properties. + +For example, in order to use the ``MyKmsClient`` defined above: + +.. code-block:: python + + >>> def kms_client_factory(kms_connection_configuration): + ... return MyKmsClient(kms_connection_configuration) + + >>> crypto_factory = pe.CryptoFactory(kms_client_factory) + +An :download:`example <../../../python/examples/parquet_encryption/sample_vault_kms_client.py>` +of such a class for an open source +`KMS <https://www.vaultproject.io/api/secret/transit>`_ can be found in the Apache +Arrow GitHub repository. The production KMS client should be designed in +cooperation with an organization's security administrators, and built by +developers with experience in access control management. Once such a class is +created, it can be passed to applications via a factory method and leveraged +by general PyArrow users as shown in the encrypted parquet write/read sample +above. + +KMS connection configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configuration of connection to KMS (:class:`pyarrow.parquet.encryption.KmsConnectionConfig` +used when creating file encryption and decryption properties) includes the +following options: + +* ``kms_instance_url``, URL of the KMS instance. +* ``kms_instance_id``, ID of the KMS instance that will be used for encryption + (if multiple KMS instances are available). +* ``key_access_token``, authorization token that will be passed to KMS. +* ``custom_kms_conf``, a string dictionary with KMS-type-specific configuration. + +Encryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.EncryptionConfiguration` (used when +creating file encryption properties) includes the following options: + +* ``footer_key``, the ID of the master key for footer encryption/signing. +* ``column_keys``, which columns to encrypt with which key. Dictionary with + master key IDs as the keys, and column name lists as the values, + e.g. ``{key1: [col1, col2], key2: [col3]}``. See notes on nested fields below. +* ``uniform_encryption``, whether to encrypt the footer and all columns with + the same ``footer_key``, instead of specifying ``column_keys`` + individually. Cannot be used together with ``column_keys``. +* ``encryption_algorithm``, the Parquet encryption algorithm. + Can be ``AES_GCM_V1`` (default) or ``AES_GCM_CTR_V1``. +* ``plaintext_footer``, whether to write the file footer in plain text (otherwise it is encrypted). +* ``double_wrapping``, whether to use double wrapping - where data encryption keys (DEKs) + are encrypted with key encryption keys (KEKs), which in turn are encrypted + with master encryption keys (MEKs). If set to ``false``, single wrapping is + used - where DEKs are encrypted directly with MEKs. +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, + local wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. +* ``internal_key_material``, whether to store key material inside Parquet file footers; + this mode doesn’t produce additional files. If set to ``false``, key material is + stored in separate files in the same folder, which enables key rotation for + immutable Parquet files. +* ``data_key_length_bits``, the length of data encryption keys (DEKs), randomly + generated by Parquet key management tools. Can be 128, 192 or 256 bits. + +.. note:: + When ``double_wrapping`` is true, Parquet implements a "double envelope + encryption" mode that minimizes the interaction of the program with a KMS + server. In this mode, the DEKs are encrypted with "key encryption keys" + (KEKs, randomly generated by Parquet). The KEKs are encrypted with "master + encryption keys" (MEKs) in the KMS; the result and the KEK itself are + cached in the process memory. + +An example encryption configuration: + +.. code-block:: python + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_name": ["Column1", "Column2"], + ... }, + ... ) + +.. note:: + + Columns with nested fields (struct or map data types) can be encrypted as a whole, or only + individual fields. Configure an encryption key for the root column name to encrypt all nested + fields with this key, or configure a key for individual leaf nested fields. + + Conventionally, the key and value fields of a map column ``m`` have the names + ``m.key_value.key`` and ``m.key_value.value``, respectively. + An inner field ``f`` of a struct column ``s`` has the name ``s.f``. + + With above example, *all* inner fields are encrypted with the same key by configuring that key + for column ``m`` and ``s``, respectively. + +An example encryption configuration for columns with nested fields, where +all columns are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + >>> import pyarrow as pa + >>> schema = pa.schema([ + ... ("MapColumn", pa.map_(pa.string(), pa.int32())), + ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ... ]) + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_id": [ "MapColumn", "StructColumn" ], + ... }, + ... ) + +An example encryption configuration for columns with nested fields, where +some inner fields are encrypted with the same key identified by ``column_key_id``: + +.. code-block:: python + + >>> schema = pa.schema([ + ... ("MapColumn", pa.map_(pa.string(), pa.int32())), + ... ("StructColumn", pa.struct([("f1", pa.int32()), ("f2", pa.string())])), + ... ]) + + >>> encryption_config = pe.EncryptionConfiguration( + ... footer_key="footer_key_name", + ... column_keys={ + ... "column_key_id": [ "MapColumn.key_value.value", "StructColumn.f1" ], + ... }, + ... ) + +Decryption configuration +~~~~~~~~~~~~~~~~~~~~~~~~ + +:class:`pyarrow.parquet.encryption.DecryptionConfiguration` (used when creating +file decryption properties) is optional and it includes the following options: + +* ``cache_lifetime``, the lifetime of cached entities (key encryption keys, local + wrapping keys, KMS client objects) represented as a ``datetime.timedelta``. + +External key material and key rotation +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When ``internal_key_material=False`` is set on ``EncryptionConfiguration``, +key material is stored in a separate file next to the Parquet file instead of +in its footer. + +Storing key material externally is what enables key rotation: +:meth:`~.crypto_factory.rotate_master_keys()` re-wraps the data encryption keys of +a file that uses external key material using new master keys and overwrites the external Review Comment: The method cross-reference `:meth:`~.crypto_factory.rotate_master_keys()`` points to a local variable name and is unlikely to resolve in Sphinx. It should reference the `CryptoFactory.rotate_master_keys` method instead. ########## docs/source/python/parquet/parquet_type_handling.rst: ########## @@ -0,0 +1,156 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. currentmodule:: pyarrow +.. _parquet_type_handling: + + +Data Type Handling +================== + +Reading types as DictionaryArray +-------------------------------- + +The ``read_dictionary`` option in ``read_table`` and ``ParquetDataset`` will +cause columns to be read as ``DictionaryArray``, which will become +``pandas.Categorical`` when converted to pandas. This option is only valid for +string and binary column types, and it can yield significantly lower memory use +and improved performance for columns with many repeated string values. + +.. code-block:: python + + >>> import pyarrow as pa + >>> import pyarrow.parquet as pq + + >>> table = pa.table({'one': [-1, None, 2.5], + ... 'two': ['foo', 'bar', 'baz'], + ... 'three': [True, False, True]}) + ... + >>> pq.write_table(table, 'example.parquet') + + >>> pq.read_table('example.parquet', read_dictionary=['two']) + pyarrow.Table + one: double + two: dictionary<values=string, indices=int32, ordered=0> + three: bool + ---- + one: [[-1,null,2.5]] + two: [ -- dictionary: + ["foo","bar","baz"] -- indices: + [0,1,2]] + three: [[true,false,true]] + +Reading binary and list columns +------------------------------- + +By default, Parquet ``BYTE_ARRAY`` columns are read as Arrow ``binary`` type +and Parquet ``LIST`` columns are read as Arrow ``list`` type, both of which use +32-bit offsets. For very large datasets this can overflow. The ``binary_type`` +and ``list_type`` parameters let you choose a different Arrow type on read. + +``binary_type`` accepts ``pa.binary()`` (default), ``pa.large_binary()``, or +``pa.binary_view()``: + +.. code-block:: python + + >>> import pyarrow as pa + >>> table = pa.table({'data': pa.array([b'hello', b'world'], pa.binary())}) + >>> pq.write_table(table, 'binary.parquet', store_schema=False) + >>> pq.read_table('binary.parquet', binary_type=pa.large_binary()).schema Review Comment: This code block uses `pq.write_table` / `pq.read_table` but does not import `pyarrow.parquet as pq`, so the example won’t run if copied independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
