This is an automated email from the ASF dual-hosted git repository.
kou pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 1e45e182a7 GH-38837: [Format] Add the specification for statistics
schema (#45058)
1e45e182a7 is described below
commit 1e45e182a71d51223cfb82b236dde07b436f44f1
Author: Sutou Kouhei <[email protected]>
AuthorDate: Tue Dec 31 17:11:41 2024 +0900
GH-38837: [Format] Add the specification for statistics schema (#45058)
### Rationale for this change
Statistics are useful for fast query processing. Many query engines
use statistics to optimize their query plan.
Apache Arrow format doesn't have statistics but other formats that can
be read as Apache Arrow data may have statistics. For example, Apache
Parquet C++ can read Apache Parquet file as Apache Arrow data and
Apache Parquet file may have statistics.
One of the Apache Arrow C streaming interface use cases is the following:
1. Module A reads Apache Parquet file as Apache Arrow data
2. Module A passes the read Apache Arrow data to module B through the
Arrow C data interface
3. Module B processes the passed Apache Arrow data
If module A can pass the statistics associated with the Apache Parquet
file to module B, module B can use the statistics to optimize its
query plan.
### What changes are included in this PR?
We standardize how to represent statistics as an Apache Arrow array
for easy to exchange.
We don't standardize how to pass the statistics array. You can use any
interface for it. For example, you can us ethe Apache Arrow C data
interface.
### Are these changes tested?
Yes.
### Are there any user-facing changes?
Yes.
* GitHub Issue: #38837
Lead-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Sutou Kouhei <[email protected]>
Co-authored-by: Felipe Oliveira Carvalho <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
---
docs/source/format/Columnar.rst | 1 +
docs/source/format/StatisticsSchema.rst | 873 ++++++++++++++++++++++++++++++++
docs/source/format/index.rst | 1 +
3 files changed, 875 insertions(+)
diff --git a/docs/source/format/Columnar.rst b/docs/source/format/Columnar.rst
index 33c937ea34..9ef6a93352 100644
--- a/docs/source/format/Columnar.rst
+++ b/docs/source/format/Columnar.rst
@@ -1619,6 +1619,7 @@ example as above, an alternate encoding could be: ::
0
EOS
+.. _format_metadata:
Custom Application Metadata
---------------------------
diff --git a/docs/source/format/StatisticsSchema.rst
b/docs/source/format/StatisticsSchema.rst
new file mode 100644
index 0000000000..01cc0da7c4
--- /dev/null
+++ b/docs/source/format/StatisticsSchema.rst
@@ -0,0 +1,873 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _statistics-schema:
+
+=================
+Statistics schema
+=================
+
+.. warning:: This specification should be considered experimental.
+
+Rationale
+=========
+
+Statistics are useful for fast query processing. Many query engines
+use statistics to optimize their query plan.
+
+Apache Arrow format doesn't have statistics but other formats that can
+be read as Apache Arrow data may have statistics. For example, the
+Apache Parquet C++ implementation can read an Apache Parquet file as
+Apache Arrow data and the Apache Parquet file may have statistics.
+
+We standardize the representation of statistics as an Apache Arrow
+array for ease of exchange.
+
+Use case
+--------
+
+One of :ref:`c-stream-interface` use cases is the following:
+
+1. Module A reads Apache Parquet file as Apache Arrow data.
+2. Module A passes the read Apache Arrow data to module B through the
+ Arrow C stream interface.
+3. Module B processes the passed Apache Arrow data.
+
+If module A can pass the statistics associated with the Apache Parquet
+file to module B, module B can use the statistics to optimize its
+query plan.
+
+For example, DuckDB uses this approach but DuckDB couldn't use
+statistics because there wasn't a standardized way to represent
+statistics for the Apache Arrow data.
+
+.. seealso::
+
+ `duckdb::ArrowTableFunction::ArrowScanBind() in DuckDB 1.1.3
+
<https://github.com/duckdb/duckdb/blob/v1.1.3/src/function/table/arrow.cpp#L373-L403>`_
+
+Goals
+-----
+
+* Establish a standard way to represent statistics as an Apache Arrow
+ array.
+
+Non-goals
+---------
+
+* Establish a standard way to pass an Apache Arrow array that
+ represents statistics.
+* Establish a standard way to embed statistics into an Apache Arrow
+ array itself.
+
+Schema
+======
+
+This specification provides only the schema for statistics. This is
+the canonical schema to represent statistics about an Apache Arrow
+dataset as Apache Arrow data.
+
+Here is the outline of the schema for statistics::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<values: utf8, indices: int32>,
+ items: dense_union<...all needed types...>
+ >
+ >
+
+Here is the details of top-level ``struct``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Nullable
+ - Notes
+ * - ``column``
+ - ``int32``
+ - ``true``
+ - The zero-based column index, or null if the statistics
+ describe the whole table or record batch.
+
+ The column index is computed as the same rule used by
+ :ref:`ipc-recordbatch-message`.
+ * - ``statistics``
+ - ``map``
+ - ``false``
+ - Statistics for the target column, table or record batch. See
+ the separate table below for details.
+
+Here is the details of the ``map`` of the ``statistics``:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Key or items
+ - Data type
+ - Nullable
+ - Notes
+ * - key
+ - ``dictionary<values: utf8, indices: int32>``
+ - ``false``
+ - The string key is the name of the
+ statistic. Dictionary-encoding is used for efficiency as the
+ same statistic may be repeated for different columns.
+ Different keys are assigned for exact and approximate statistic
+ values. Each statistic has their own description below.
+ * - items
+ - ``dense_union``
+ - ``false``
+ - Statistics value is dense union. It has at least all needed
+ types based on statistics kinds in the keys. For example, you
+ need at least ``int64`` and ``float64`` types when you have a
+ ``int64`` distinct count statistic and a ``float64`` average
+ byte width statistic. See the description of each statistic below.
+
+ Dense union arrays have names for each field but we don't standardize
+ field names for these because we can access the proper
+ field by type code instead. So we can use any valid name for
+ the fields.
+
+.. _statistics-schema-name:
+
+Standard statistics
+-------------------
+
+Each statistic kind has a name that appears as a key in the statistics
+map for each column or entire table. ``dictionary<values: utf8,
+indices: int32>`` is used to encode the name for space-efficiency.
+
+We assign different names for variations of the same statistic instead
+of using flags. For example, we assign different statistic names for
+exact and approximate values of the "distinct count" statistic.
+
+The colon symbol ``:`` is to be used as a namespace separator like
+:ref:`format_metadata`. It can be used multiple times in a name.
+
+The ``ARROW`` prefix is a reserved namespace for pre-defined statistic
+names in current and future versions of this specification.
+User-defined statistics must not use it. For example, you can use your
+product name as namespace such as ``MY_PRODUCT:my_statistics:exact``.
+
+Here are pre-defined statistics names:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Name
+ - Data type
+ - Notes
+ * - ``ARROW:average_byte_width:exact``
+ - ``float64``
+ - The average size in bytes of a row in the target
+ column. (exact)
+ * - ``ARROW:average_byte_width:approximate``
+ - ``float64``
+ - The average size in bytes of a row in the target
+ column. (approximate)
+ * - ``ARROW:distinct_count:exact``
+ - ``int64``
+ - The number of distinct values in the target column. (exact)
+ * - ``ARROW:distinct_count:approximate``
+ - ``float64``
+ - The number of distinct values in the target
+ column. (approximate)
+ * - ``ARROW:max_byte_width:exact``
+ - ``int64``
+ - The maximum size in bytes of a row in the target
+ column. (exact)
+ * - ``ARROW:max_byte_width:approximate``
+ - ``float64``
+ - The maximum size in bytes of a row in the target
+ column. (approximate)
+ * - ``ARROW:max_value:exact``
+ - Target dependent
+ - The maximum value in the target column. (exact)
+ * - ``ARROW:max_value:approximate``
+ - Target dependent
+ - The maximum value in the target column. (approximate)
+ * - ``ARROW:min_value:exact``
+ - Target dependent
+ - The minimum value in the target column. (exact)
+ * - ``ARROW:min_value:approximate``
+ - Target dependent
+ - The minimum value in the target column. (approximate)
+ * - ``ARROW:null_count:exact``
+ - ``int64``
+ - The number of nulls in the target column. (exact)
+ * - ``ARROW:null_count:approximate``
+ - ``float64``
+ - The number of nulls in the target column. (approximate)
+ * - ``ARROW:row_count:exact``
+ - ``int64``
+ - The number of rows in the target table, record batch or
+ array. (exact)
+ * - ``ARROW:row_count:approximate``
+ - ``float64``
+ - The number of rows in the target table, record batch or
+ array. (approximate)
+
+If you find a statistic that might be useful to multiple systems,
+please propose it on the `Apache Arrow development mailing-list
+<https://arrow.apache.org/community/>`__.
+
+Interoperability improves when producers and consumers of statistics
+follow a previously agreed upon statistic specification.
+
+.. _statistics-schema-examples:
+
+Examples
+========
+
+Here are some examples to help you understand.
+
+Simple record batch
+-------------------
+
+Schema::
+
+ vendor_id: int32
+ passenger_count: int64
+
+Data::
+
+ vendor_id: [5, 1, 5, 1, 5]
+ passenger_count: [1, 1, 2, 0, null]
+
+Statistics:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Target
+ - Name
+ - Value
+ * - Record batch
+ - The number of rows
+ - ``5``
+ * - ``vendor_id``
+ - The number of nulls
+ - ``0``
+ * - ``vendor_id``
+ - The number of distinct values
+ - ``2``
+ * - ``vendor_id``
+ - The max value
+ - ``5``
+ * - ``vendor_id``
+ - The min value
+ - ``1``
+ * - ``passenger_count``
+ - The number of nulls
+ - ``1``
+ * - ``passenger_count``
+ - The number of distinct values
+ - ``3``
+ * - ``passenger_count``
+ - The max value
+ - ``2``
+ * - ``passenger_count``
+ - The min value
+ - ``0``
+
+Column indexes:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Index
+ - Target
+ * - ``0``
+ - ``vendor_id``
+ * - ``1``
+ - ``passenger_count``
+
+Statistics schema::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<values: utf8, indices: int32>,
+ items: dense_union<0: int64>
+ >
+ >
+
+Statistics array::
+
+ column: [
+ null, # record batch
+ 0, # vendor_id
+ 0, # vendor_id
+ 0, # vendor_id
+ 0, # vendor_id
+ 1, # passenger_count
+ 1, # passenger_count
+ 1, # passenger_count
+ 1, # passenger_count
+ ]
+ statistics:
+ key:
+ values: [
+ "ARROW:row_count:exact",
+ "ARROW:null_count:exact",
+ "ARROW:distinct_count:exact",
+ "ARROW:max_value:exact",
+ "ARROW:min_value:exact",
+ ],
+ indices: [
+ 0, # "ARROW:row_count:exact"
+ 1, # "ARROW:null_count:exact"
+ 2, # "ARROW:distinct_count:exact"
+ 3, # "ARROW:max_value:exact"
+ 4, # "ARROW:min_value:exact"
+ 1, # "ARROW:null_count:exact"
+ 2, # "ARROW:distinct_count:exact"
+ 3, # "ARROW:max_value:exact"
+ 4, # "ARROW:min_value:exact"
+ ]
+ items:
+ children:
+ 0: [ # int64
+ 5, # record batch: "ARROW:row_count:exact"
+ 0, # vendor_id: "ARROW:null_count:exact"
+ 2, # vendor_id: "ARROW:distinct_count:exact"
+ 5, # vendor_id: "ARROW:max_value:exact"
+ 1, # vendor_id: "ARROW:min_value:exact"
+ 1, # passenger_count: "ARROW:null_count:exact"
+ 3, # passenger_count: "ARROW:distinct_count:exact"
+ 2, # passenger_count: "ARROW:max_value:exact"
+ 0, # passenger_count: "ARROW:min_value:exact"
+ ]
+ types: [ # all values are int64
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ ]
+ offsets: [
+ 0,
+ 1,
+ 2,
+ 3,
+ 4,
+ 5,
+ 6,
+ 7,
+ 8,
+ ]
+
+Complex record batch
+--------------------
+
+This uses nested types.
+
+Schema::
+
+ col1: struct<a: int32, b: list<item: int64>, c: float64>
+ col2: utf8
+
+Data::
+
+ col1: [
+ {a: 1, b: [20, 30, 40], c: 2.9},
+ {a: 2, b: null, c: -2.9},
+ {a: 3, b: [99], c: null},
+ ]
+ col2: ["x", null, "z"]
+
+Statistics:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Target
+ - Name
+ - Value
+ * - Record batch
+ - The number of rows
+ - ``3``
+ * - ``col1``
+ - The number of nulls
+ - ``0``
+ * - ``col1.a``
+ - The number of nulls
+ - ``0``
+ * - ``col1.a``
+ - The number of distinct values
+ - ``3``
+ * - ``col1.a``
+ - The approximate max value
+ - ``5``
+ * - ``col1.a``
+ - The approximate min value
+ - ``0``
+ * - ``col1.b``
+ - The number of nulls
+ - ``1``
+ * - ``col1.b.item``
+ - The max value
+ - ``99``
+ * - ``col1.b.item``
+ - The min value
+ - ``20``
+ * - ``col1.c``
+ - The number of nulls
+ - ``1``
+ * - ``col1.c``
+ - The approximate max value
+ - ``3.0``
+ * - ``col1.c``
+ - The approximate min value
+ - ``-3.0``
+ * - ``col2``
+ - The number of nulls
+ - ``1``
+ * - ``col2``
+ - The number of distinct values
+ - ``2``
+
+Column indexes:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Index
+ - Target
+ * - ``0``
+ - ``col1``
+ * - ``1``
+ - ``col1.a``
+ * - ``2``
+ - ``col1.b``
+ * - ``3``
+ - ``col1.b.item``
+ * - ``4``
+ - ``col1.c``
+ * - ``5``
+ - ``col2``
+
+See also :ref:`ipc-recordbatch-message` how to compute column indexes.
+
+Statistics schema::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<values: utf8, indices: int32>,
+ items: dense_union<
+ # For the number of rows, the number of nulls and so on.
+ 0: int64,
+ # For the max/min values of col1.c.
+ 1: float64
+ >
+ >
+ >
+
+Statistics array::
+
+ column: [
+ null, # record batch
+ 0, # col1
+ 1, # col1.a
+ 1, # col1.a
+ 1, # col1.a
+ 1, # col1.a
+ 2, # col1.b
+ 3, # col1.b.item
+ 3, # col1.b.item
+ 4, # col1.c
+ 4, # col1.c
+ 4, # col1.c
+ 5, # col2
+ 5, # col2
+ ]
+ statistics:
+ key:
+ values: [
+ "ARROW:row_count:exact",
+ "ARROW:null_count:exact",
+ "ARROW:distinct_count:exact",
+ "ARROW:max_value:approximate",
+ "ARROW:min_value:approximate",
+ "ARROW:max_value:exact",
+ "ARROW:min_value:exact",
+ ]
+ indices: [
+ 0, # "ARROW:row_count:exact"
+ 1, # "ARROW:null_count:exact"
+ 1, # "ARROW:null_count:exact"
+ 2, # "ARROW:distinct_count:exact"
+ 3, # "ARROW:max_value:approximate"
+ 4, # "ARROW:min_value:approximate"
+ 1, # "ARROW:null_count:exact"
+ 5, # "ARROW:max_value:exact"
+ 6, # "ARROW:min_value:exact"
+ 1, # "ARROW:null_count:exact"
+ 3, # "ARROW:max_value:approximate"
+ 4, # "ARROW:min_value:approximate"
+ 1, # "ARROW:null_count:exact"
+ 2, # "ARROW:distinct_count:exact"
+ ]
+ items:
+ children:
+ 0: [ # int64
+ 3, # record batch: "ARROW:row_count:exact"
+ 0, # col1: "ARROW:null_count:exact"
+ 0, # col1.a: "ARROW:null_count:exact"
+ 3, # col1.a: "ARROW:distinct_count:exact"
+ 5, # col1.a: "ARROW:max_value:approximate"
+ 0, # col1.a: "ARROW:min_value:approximate"
+ 1, # col1.b: "ARROW:null_count:exact"
+ 99, # col1.b.item: "ARROW:max_value:exact"
+ 20, # col1.b.item: "ARROW:min_value:exact"
+ 1, # col1.c: "ARROW:null_count:exact"
+ 1, # col2: "ARROW:null_count:exact"
+ 2, # col2: "ARROW:distinct_count:exact"
+ ]
+ 1: [ # float64
+ 3.0, # col1.c: "ARROW:max_value:approximate"
+ -3.0, # col1.c: "ARROW:min_value:approximate"
+ ]
+ types: [
+ 0, # int64: record batch: "ARROW:row_count:exact"
+ 0, # int64: col1: "ARROW:null_count:exact"
+ 0, # int64: col1.a: "ARROW:null_count:exact"
+ 0, # int64: col1.a: "ARROW:distinct_count:exact"
+ 0, # int64: col1.a: "ARROW:max_value:approximate"
+ 0, # int64: col1.a: "ARROW:min_value:approximate"
+ 0, # int64: col1.b: "ARROW:null_count:exact"
+ 0, # int64: col1.b.item: "ARROW:max_value:exact"
+ 0, # int64: col1.b.item: "ARROW:min_value:exact"
+ 0, # int64: col1.c: "ARROW:null_count:exact"
+ 1, # float64: col1.c: "ARROW:max_value:approximate"
+ 1, # float64: col1.c: "ARROW:min_value:approximate"
+ 0, # int64: col2: "ARROW:null_count:exact"
+ 0, # int64: col2: "ARROW:distinct_count:exact"
+ ]
+ offsets: [
+ 0, # int64: record batch: "ARROW:row_count:exact"
+ 1, # int64: col1: "ARROW:null_count:exact"
+ 2, # int64: col1.a: "ARROW:null_count:exact"
+ 3, # int64: col1.a: "ARROW:distinct_count:exact"
+ 4, # int64: col1.a: "ARROW:max_value:approximate"
+ 5, # int64: col1.a: "ARROW:min_value:approximate"
+ 6, # int64: col1.b: "ARROW:null_count:exact"
+ 7, # int64: col1.b.item: "ARROW:max_value:exact"
+ 8, # int64: col1.b.item: "ARROW:min_value:exact"
+ 9, # int64: col1.c: "ARROW:null_count:exact"
+ 0, # float64: col1.c: "ARROW:max_value:approximate"
+ 1, # float64: col1.c: "ARROW:min_value:approximate"
+ 10, # int64: col2: "ARROW:null_count:exact"
+ 11, # int64: col2: "ARROW:distinct_count:exact"
+ ]
+
+Simple array
+------------
+
+Schema::
+
+ int64
+
+Data::
+
+ [1, 1, 2, 0, null]
+
+Statistics:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Target
+ - Name
+ - Value
+ * - Array
+ - The number of rows
+ - ``5``
+ * - Array
+ - The number of nulls
+ - ``1``
+ * - Array
+ - The number of distinct values
+ - ``3``
+ * - Array
+ - The max value
+ - ``2``
+ * - Array
+ - The min value
+ - ``0``
+
+Column indexes:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Index
+ - Target
+ * - ``0``
+ - Array
+
+Statistics schema::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<values: utf8, indices: int32>,
+ items: dense_union<0: int64>
+ >
+ >
+
+Statistics array::
+
+ column: [
+ 0, # array
+ 0, # array
+ 0, # array
+ 0, # array
+ 0, # array
+ ]
+ statistics:
+ key:
+ values: [
+ "ARROW:row_count:exact",
+ "ARROW:null_count:exact",
+ "ARROW:distinct_count:exact",
+ "ARROW:max_value:exact",
+ "ARROW:min_value:exact",
+ ]
+ indices: [
+ 0, # "ARROW:row_count:exact"
+ 1, # "ARROW:null_count:exact"
+ 2, # "ARROW:distinct_count:exact"
+ 3, # "ARROW:max_value:exact"
+ 4, # "ARROW:min_value:exact"
+ ]
+ items:
+ children:
+ 0: [ # int64
+ 5, # array: "ARROW:row_count:exact"
+ 1, # array: "ARROW:null_count:exact"
+ 3, # array: "ARROW:distinct_count:exact"
+ 2, # array: "ARROW:max_value:exact"
+ 0, # array: "ARROW:min_value:exact"
+ ]
+ types: [ # all values are int64
+ 0,
+ 0,
+ 0,
+ 0,
+ 0,
+ ]
+ offsets: [
+ 0,
+ 1,
+ 2,
+ 3,
+ 4,
+ ]
+
+Complex array
+-------------
+
+This uses nested types.
+
+Schema::
+
+ struct<a: int32, b: list<item: int64>, c: float64>
+
+Data::
+
+ [
+ {a: 1, b: [20, 30, 40], c: 2.9},
+ {a: 2, b: null, c: -2.9},
+ {a: 3, b: [99], c: null},
+ ]
+
+Statistics:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Target
+ - Name
+ - Value
+ * - Array
+ - The number of rows
+ - ``3``
+ * - Array
+ - The number of nulls
+ - ``0``
+ * - ``a``
+ - The number of nulls
+ - ``0``
+ * - ``a``
+ - The number of distinct values
+ - ``3``
+ * - ``a``
+ - The approximate max value
+ - ``5``
+ * - ``a``
+ - The approximate min value
+ - ``0``
+ * - ``b``
+ - The number of nulls
+ - ``1``
+ * - ``b.item``
+ - The max value
+ - ``99``
+ * - ``b.item``
+ - The min value
+ - ``20``
+ * - ``c``
+ - The number of nulls
+ - ``1``
+ * - ``c``
+ - The approximate max value
+ - ``3.0``
+ * - ``c``
+ - The approximate min value
+ - ``-3.0``
+
+Column indexes:
+
+.. list-table::
+ :header-rows: 1
+
+ * - Index
+ - Target
+ * - ``0``
+ - Array
+ * - ``1``
+ - ``a``
+ * - ``2``
+ - ``b``
+ * - ``3``
+ - ``b.item``
+ * - ``4``
+ - ``c``
+
+See also :ref:`ipc-recordbatch-message` how to compute column indexes.
+
+Statistics schema::
+
+ struct<
+ column: int32,
+ statistics: map<
+ key: dictionary<values: utf8, indices: int32>,
+ items: dense_union<
+ # For the number of rows, the number of nulls and so on.
+ 0: int64,
+ # For the max/min values of c.
+ 1: float64
+ >
+ >
+ >
+
+Statistics array::
+
+ column: [
+ 0, # array
+ 0, # array
+ 1, # a
+ 1, # a
+ 1, # a
+ 1, # a
+ 2, # b
+ 3, # b.item
+ 3, # b.item
+ 4, # c
+ 4, # c
+ 4, # c
+ ]
+ statistics:
+ key:
+ values: [
+ "ARROW:row_count:exact",
+ "ARROW:null_count:exact",
+ "ARROW:distinct_count:exact",
+ "ARROW:max_value:approximate",
+ "ARROW:min_value:approximate",
+ "ARROW:max_value:exact",
+ "ARROW:min_value:exact",
+ ]
+ indices: [
+ 0, # "ARROW:row_count:exact"
+ 1, # "ARROW:null_count:exact"
+ 1, # "ARROW:null_count:exact"
+ 2, # "ARROW:distinct_count:exact"
+ 3, # "ARROW:max_value:approximate"
+ 4, # "ARROW:min_value:approximate"
+ 1, # "ARROW:null_count:exact"
+ 5, # "ARROW:max_value:exact"
+ 6, # "ARROW:min_value:exact"
+ 1, # "ARROW:null_count:exact"
+ 3, # "ARROW:max_value:approximate"
+ 4, # "ARROW:min_value:approximate"
+ ]
+ items:
+ children:
+ 0: [ # int64
+ 3, # array: "ARROW:row_count:exact"
+ 0, # array: "ARROW:null_count:exact"
+ 0, # a: "ARROW:null_count:exact"
+ 3, # a: "ARROW:distinct_count:exact"
+ 5, # a: "ARROW:max_value:approximate"
+ 0, # a: "ARROW:min_value:approximate"
+ 1, # b: "ARROW:null_count:exact"
+ 99, # b.item: "ARROW:max_value:exact"
+ 20, # b.item: "ARROW:min_value:exact"
+ 1, # c: "ARROW:null_count:exact"
+ ]
+ 1: [ # float64
+ 3.0, # c: "ARROW:max_value:approximate"
+ -3.0, # c: "ARROW:min_value:approximate"
+ ]
+ types: [
+ 0, # int64: array: "ARROW:row_count:exact"
+ 0, # int64: array: "ARROW:null_count:exact"
+ 0, # int64: a: "ARROW:null_count:exact"
+ 0, # int64: a: "ARROW:distinct_count:exact"
+ 0, # int64: a: "ARROW:max_value:approximate"
+ 0, # int64: a: "ARROW:min_value:approximate"
+ 0, # int64: b: "ARROW:null_count:exact"
+ 0, # int64: b.item: "ARROW:max_value:exact"
+ 0, # int64: b.item: "ARROW:min_value:exact"
+ 0, # int64: c: "ARROW:null_count:exact"
+ 1, # float64: c: "ARROW:max_value:approximate"
+ 1, # float64: c: "ARROW:min_value:approximate"
+ ]
+ offsets: [
+ 0, # int64: array: "ARROW:row_count:exact"
+ 1, # int64: array: "ARROW:null_count:exact"
+ 2, # int64: a: "ARROW:null_count:exact"
+ 3, # int64: a: "ARROW:distinct_count:exact"
+ 4, # int64: a: "ARROW:max_value:approximate"
+ 5, # int64: a: "ARROW:min_value:approximate"
+ 6, # int64: b: "ARROW:null_count:exact"
+ 7, # int64: b.item: "ARROW:max_value:exact"
+ 8, # int64: b.item: "ARROW:min_value:exact"
+ 9, # int64: c: "ARROW:null_count:exact"
+ 0, # float64: c: "ARROW:max_value:approximate"
+ 1, # float64: c: "ARROW:min_value:approximate"
+ ]
diff --git a/docs/source/format/index.rst b/docs/source/format/index.rst
index ce31a15a1f..91912a5325 100644
--- a/docs/source/format/index.rst
+++ b/docs/source/format/index.rst
@@ -32,6 +32,7 @@ Specifications
CDataInterface
CStreamInterface
CDeviceDataInterface
+ StatisticsSchema
DissociatedIPC
Flight
FlightSql
