kou commented on code in PR #43553: URL: https://github.com/apache/arrow/pull/43553#discussion_r1872626610
########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. Review Comment: I missed the nested case. How about using the same rule used by the RecordBatch message? https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So Review Comment: Ah, yes. You're right. I should have used "by type code" not "by index". ```suggestion consumers can access to proper field by type code not name. So ``` ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact Review Comment: "instead of using flags" means that we use "max" statistics key + "exact" flag for exact max statistics and "max" statistics key + "approximate" flag for approximate max statistics. ADBC uses this style: https://github.com/apache/arrow-adbc/blob/95f55c73e916a9b47bc287d10296e8f561284d42/c/include/arrow-adbc/adbc.h#L1761 ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact +value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. + +Here are pre-defined statistics keys: + +.. list-table:: + :header-rows: 1 + + * - Key + - Data type + - Notes + * - ``ARROW:average_byte_width:exact`` + - ``float`` + - The average size in bytes of a row in the target. (exact) + * - ``ARROW:average_byte_width:approximate`` + - ``float64`` + - The average size in bytes of a row in the target. (approximate) + * - ``ARROW:distinct_count:exact`` + - ``int64`` + - The number of distinct values in the target. (exact) + * - ``ARROW:distinct_count:approximate`` + - ``float64`` + - The number of distinct values in the target. (approximate) + * - ``ARROW:max_byte_width:exact`` + - ``int64`` + - The maximum size in bytes of a row in the target. (exact) + * - ``ARROW:max_byte_width:approximate`` + - ``float64`` + - The maximum size in bytes of a row in the target. (approximate) + * - ``ARROW:max_value:exact`` + - Target dependent + - The maximum value in the target. (exact) + * - ``ARROW:max_value:approximate`` + - Target dependent + - The maximum value in the target. (approximate) + * - ``ARROW:min_value:exact`` + - Target dependent + - The minimum value in the target. (exact) + * - ``ARROW:min_value:approximate`` + - Target dependent + - The minimum value in the target. (approximate) + * - ``ARROW:null_count:exact`` + - ``int64`` + - The number of nulls in the target. (exact) + * - ``ARROW:null_count:approximate`` + - ``float64`` + - The number of nulls in the target. (approximate) + * - ``ARROW:row_count:exact`` + - ``int64`` + - The number of rows in the target table or record batch. (exact) Review Comment: It's not known yet. In general, a statistics array is passed before we receive a target record batch. We don't know the number of rows in a target record batch at the time. ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, Review Comment: Good point. I missed nested type case. How about using the same rule used by the RecordBatch message? https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message It may be better that we use `field` not `column` like RecordBatch message does. ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface Review Comment: Right. But I'm not sure whether this approach is the best approach not for the Arrow C data interface... See also: https://github.com/apache/arrow/pull/43553/files#r1704373291 I think that our discussion focused on only the Arrow C data interface: https://lists.apache.org/thread/6m9xrhfktnt0nnyss7qo333n0cl76ypc > Out of scope: > > * Transmit statistics through not the C data interface > > Examples: > * Transmit statistics through Apache Arrow IPC file > * Transmit statistics through Apache Arrow Flight With the C data interface, we have only limited approaches for this. For example, separated Arrow array like this approach, metadata in `ArrowSchema` and so on. But we may have more approaches with other interfaces. Other approaches may be better than this approach with other approaches. I'm OK with expanding scope of this specification. Is it OK that we restart this discussion with not only the C data interface but also all interfaces? ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using Review Comment: Yes. I think that I borrowed "variants" from a suggestion from others but I can't find it... Let's use "indivisual statistics": ```suggestion We assign different statistics keys for individual statistics instead of using ``` ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact +value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. Review Comment: It makes sense: ```suggestion statistics keys. User-defined statistics must not use it. For example, you can use your product name as namespace such as `MY_PRODUCT:my_statistics:exact`. ``` ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. Review Comment: @lidavidm Do you have any opinion for unifying this specification and the ADBC's statistics related APIs? ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. Review Comment: Hmm. I think that it's difficult that we standardize field names. Because we don't define concrete field types in this specification. We can use `str(type_code)` for field name. Does it make sense? ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact +value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. + +Here are pre-defined statistics keys: + +.. list-table:: + :header-rows: 1 + + * - Key + - Data type + - Notes + * - ``ARROW:average_byte_width:exact`` + - ``float`` + - The average size in bytes of a row in the target. (exact) Review Comment: Right. Let's clarify it: ```suggestion - The average size in bytes of a row in the target column. (exact) ``` (I'll change others like this.) ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact +value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. + +Here are pre-defined statistics keys: + +.. list-table:: + :header-rows: 1 + + * - Key + - Data type + - Notes + * - ``ARROW:average_byte_width:exact`` + - ``float`` Review Comment: Oh, I should have used "float64" like others: ```suggestion - ``float64`` ``` ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact +value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. + +Here are pre-defined statistics keys: + +.. list-table:: + :header-rows: 1 + + * - Key + - Data type + - Notes + * - ``ARROW:average_byte_width:exact`` + - ``float`` + - The average size in bytes of a row in the target. (exact) + * - ``ARROW:average_byte_width:approximate`` + - ``float64`` + - The average size in bytes of a row in the target. (approximate) + * - ``ARROW:distinct_count:exact`` + - ``int64`` + - The number of distinct values in the target. (exact) + * - ``ARROW:distinct_count:approximate`` + - ``float64`` + - The number of distinct values in the target. (approximate) + * - ``ARROW:max_byte_width:exact`` + - ``int64`` + - The maximum size in bytes of a row in the target. (exact) + * - ``ARROW:max_byte_width:approximate`` + - ``float64`` + - The maximum size in bytes of a row in the target. (approximate) + * - ``ARROW:max_value:exact`` + - Target dependent + - The maximum value in the target. (exact) + * - ``ARROW:max_value:approximate`` + - Target dependent + - The maximum value in the target. (approximate) + * - ``ARROW:min_value:exact`` + - Target dependent + - The minimum value in the target. (exact) + * - ``ARROW:min_value:approximate`` + - Target dependent + - The minimum value in the target. (approximate) + * - ``ARROW:null_count:exact`` Review Comment: It's the number of `null` for the array. In this case, the array is a statistics array not a target column/record batch. This is for a target column/record batch not a statistics array. ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact +value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. + +Here are pre-defined statistics keys: + +.. list-table:: + :header-rows: 1 + + * - Key + - Data type + - Notes + * - ``ARROW:average_byte_width:exact`` + - ``float`` + - The average size in bytes of a row in the target. (exact) + * - ``ARROW:average_byte_width:approximate`` + - ``float64`` + - The average size in bytes of a row in the target. (approximate) + * - ``ARROW:distinct_count:exact`` + - ``int64`` + - The number of distinct values in the target. (exact) + * - ``ARROW:distinct_count:approximate`` + - ``float64`` + - The number of distinct values in the target. (approximate) + * - ``ARROW:max_byte_width:exact`` + - ``int64`` + - The maximum size in bytes of a row in the target. (exact) + * - ``ARROW:max_byte_width:approximate`` + - ``float64`` Review Comment: It's correct for the "exact" case. But I think that float64 is better for the "approximate" case. Because an approximate max byte width may be float for string array. ########## docs/source/format/CDataInterfaceStatistics.rst: ########## @@ -0,0 +1,271 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. _c-data-interface-statistics: + +===================================================== +Passing statistics through the Arrow C data interface +===================================================== + +.. warning:: This specification should be considered experimental. + +Rationale +========= + +Statistics are useful for fast query processing. Many query engines +use statistics to optimize their query plan. + +Apache Arrow format doesn't have statistics but other formats that can +be read as Apache Arrow data may have statistics. For example, the +Apache Parquet C++ implementation can read an Apache Parquet file as +Apache Arrow data and the Apache Parquet file may have statistics. + +One of :ref:`c-data-interface` use cases is the following: + +1. Module A reads Apache Parquet file as Apache Arrow data. +2. Module A passes the read Apache Arrow data to module B through the + Arrow C data interface. +3. Module B processes the passed Apache Arrow data. + +If module A can pass the statistics associated with the Apache Parquet +file to module B through the Arrow C data interface, module B can use +the statistics to optimize its query plan. + +Goals +----- + +* Establish a standard way to pass statistics through the Arrow C data + interface. +* Provide this in a manner that enables compatibility and ease of + implementation for existing users of the Arrow C data interface. + +Non-goals +--------- + +* Provide a common way to pass statistics that can be used for + other interfaces such Arrow Flight too. + +For example, ADBC has `the statistics related APIs +<https://arrow.apache.org/adbc/current/format/specification.html#statistics>`__. +This specification doesn't replace them. + +This specification may fit some use cases of :ref:`format-ipc` not the +Arrow data interface. But we don't recommend this specification for +the Arrow IPC format for now. Because we may be able to define better +specification for the Arrow IPC format. The Arrow IPC format has some +different features compared with the Arrow C data interface. For +example, the Arrow IPC format can have :ref:`metadata for each message +<ipc-message-format>`. If you're interested in the specification for +passing statistics through the Arrow IPC format, please start a +discussion on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +.. _c-data-interface-statistics-schema: + +Schema +====== + +This specification provides only the schema for statistics. The +producer passes statistics through the Arrow C data interface as an +Arrow map array that uses this schema. + +Here is the outline of the schema for statistics:: + + struct< + column: int32, + statistics: map< + key: dictionary< + indices: int32, + dictionary: utf8 + >, + items: dense_union<...all needed types...>, + > + > + +Here is the details of top-level ``struct``: + +.. list-table:: + :header-rows: 1 + + * - Name + - Data type + - Nullable + - Notes + * - ``column`` + - ``int32`` + - ``true`` + - The zero-based column index, or null if the statistics + describe the whole table or record batch. + * - ``statistics`` + - ``map`` + - ``false`` + - Statistics for the target column, table or record batch. See + the separate table below for details. + +Here is the details of the ``map`` of the ``statistics``: + +.. list-table:: + :header-rows: 1 + + * - Key or items + - Data type + - Nullable + - Notes + * - key + - ``dictionary<indices: int32, dictionary: utf8>`` + - ``false`` + - Statistics key is string. Dictionary is used for + efficiency. Different keys are assigned for exact value and + approximate value. Also see the separate description below for + statistics key. + * - items + - ``dense_union`` + - ``false`` + - Statistics value is dense union. It has at least all needed + types based on statistics kinds in the keys. For example, you + need at least ``int64`` and ``float64`` types when you have a + ``int64`` distinct count statistic and a ``float64`` average + byte width statistic. Also see the separate description below + for statistics key. + + We don't standardize field names for the dense union because + consumers can access to proper field by index not name. So + producers can use any valid name for fields. + +.. _c-data-interface-statistics-key: + +Statistics key +-------------- + +Statistics key is string. ``dictionary<int32, utf8>`` is used for +efficiency. + +We assign different statistics keys for variants instead of using +flags. For example, we assign different statistics keys for exact +value and approximate value. + +The colon symbol ``:`` is to be used as a namespace separator like +:ref:`format_metadata`. It can be used multiple times in a key. + +The ``ARROW`` pattern is a reserved namespace for pre-defined +statistics keys. User-defined statistics must not use it. + +Here are pre-defined statistics keys: + +.. list-table:: + :header-rows: 1 + + * - Key + - Data type + - Notes + * - ``ARROW:average_byte_width:exact`` + - ``float`` + - The average size in bytes of a row in the target. (exact) + * - ``ARROW:average_byte_width:approximate`` + - ``float64`` + - The average size in bytes of a row in the target. (approximate) + * - ``ARROW:distinct_count:exact`` + - ``int64`` + - The number of distinct values in the target. (exact) + * - ``ARROW:distinct_count:approximate`` + - ``float64`` + - The number of distinct values in the target. (approximate) + * - ``ARROW:max_byte_width:exact`` + - ``int64`` + - The maximum size in bytes of a row in the target. (exact) + * - ``ARROW:max_byte_width:approximate`` + - ``float64`` + - The maximum size in bytes of a row in the target. (approximate) + * - ``ARROW:max_value:exact`` + - Target dependent + - The maximum value in the target. (exact) + * - ``ARROW:max_value:approximate`` + - Target dependent + - The maximum value in the target. (approximate) + * - ``ARROW:min_value:exact`` + - Target dependent + - The minimum value in the target. (exact) + * - ``ARROW:min_value:approximate`` + - Target dependent + - The minimum value in the target. (approximate) + * - ``ARROW:null_count:exact`` + - ``int64`` + - The number of nulls in the target. (exact) + * - ``ARROW:null_count:approximate`` + - ``float64`` + - The number of nulls in the target. (approximate) + * - ``ARROW:row_count:exact`` + - ``int64`` + - The number of rows in the target table or record batch. (exact) + * - ``ARROW:row_count:approximate`` + - ``float64`` + - The number of rows in the target table or record + batch. (approximate) + +If you find a missing statistics key that is usable for multiple +systems, please propose it on the `Arrow development mailing-list +<https://arrow.apache.org/community/>`__. + +Examples +======== + +Here are some examples to help you understand. + +C++ Review Comment: OK. I'll remove it. I'll add an example data record batch and statistics array for it instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
