alamb commented on code in PR #17126: URL: https://github.com/apache/datafusion/pull/17126#discussion_r2267909660
########## docs/source/user-guide/cli/functions.md: ########## @@ -0,0 +1,142 @@ +<!--- Review Comment: I moved the datafusion specific CLI functions to their own page ########## docs/source/user-guide/cli/functions.md: ########## @@ -0,0 +1,142 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# CLI Specific Functions + +`datafusion-cli` comes with build-in functions that are not included in the +DataFusion SQL engine by default. These functions are: + +## `parquet_metadata` + +The `parquet_metadata` table function can be used to inspect detailed metadata +about a parquet file such as statistics, sizes, and other information. This can +be helpful to understand how parquet files are structured. + +For example, to see information about the `"WatchID"` column in the +`hits.parquet` file, you can use: + +```sql +SELECT path_in_schema, row_group_id, row_group_num_rows, stats_min, stats_max, total_compressed_size +FROM parquet_metadata('hits.parquet') +WHERE path_in_schema = '"WatchID"' +LIMIT 3; + ++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+ +| path_in_schema | row_group_id | row_group_num_rows | stats_min | stats_max | total_compressed_size | ++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+ +| "WatchID" | 0 | 450560 | 4611687214012840539 | 9223369186199968220 | 3883759 | +| "WatchID" | 1 | 612174 | 4611689135232456464 | 9223371478009085789 | 5176803 | +| "WatchID" | 2 | 344064 | 4611692774829951781 | 9223363791697310021 | 3031680 | ++----------------+--------------+--------------------+---------------------+---------------------+-----------------------+ +3 rows in set. Query took 0.053 seconds. +``` + +The returned table has the following columns for each row for each column chunk +in the file. Please refer to the [Parquet Documentation] for more information in +the meaning of these fields. + +[parquet documentation]: https://parquet.apache.org/ + +| column_name | data_type | Description | +| ----------------------- | --------- | --------------------------------------------------------------------------------------------------- | +| filename | Utf8 | Name of the file | +| row_group_id | Int64 | Row group index the column chunk belongs to | +| row_group_num_rows | Int64 | Count of rows stored in the row group | +| row_group_num_columns | Int64 | Total number of columns in the row group (same for all row groups) | +| row_group_bytes | Int64 | Number of bytes used to store the row group (not including metadata) | +| column_id | Int64 | ID of the column | +| file_offset | Int64 | Offset within the file that this column chunk's data begins | +| num_values | Int64 | Total number of values in this column chunk | +| path_in_schema | Utf8 | "Path" (column name) of the column chunk in the schema | +| type | Utf8 | Parquet data type of the column chunk | +| stats_min | Utf8 | The minimum value for this column chunk, if stored in the statistics, cast to a string | +| stats_max | Utf8 | The maximum value for this column chunk, if stored in the statistics, cast to a string | +| stats_null_count | Int64 | Number of null values in this column chunk, if stored in the statistics | +| stats_distinct_count | Int64 | Number of distinct values in this column chunk, if stored in the statistics | +| stats_min_value | Utf8 | Same as `stats_min` | +| stats_max_value | Utf8 | Same as `stats_max` | +| compression | Utf8 | Block level compression (e.g. `SNAPPY`) used for this column chunk | +| encodings | Utf8 | All block level encodings (e.g. `[PLAIN_DICTIONARY, PLAIN, RLE]`) used for this column chunk | +| index_page_offset | Int64 | Offset in the file of the [`page index`], if any | +| dictionary_page_offset | Int64 | Offset in the file of the dictionary page, if any | +| data_page_offset | Int64 | Offset in the file of the first data page, if any | +| total_compressed_size | Int64 | Number of bytes the column chunk's data after encoding and compression (what is stored in the file) | +| total_uncompressed_size | Int64 | Number of bytes the column chunk's data after encoding | + +[`page index`]: https://github.com/apache/parquet-format/blob/master/PageIndex.md + +## `metadata_cache` Review Comment: Here is the new documentation for `metadata_cache` I think it looks pretty good rendered as well <img width="645" height="1129" alt="Screenshot 2025-08-11 at 4 08 45 PM" src="https://github.com/user-attachments/assets/84698260-d227-4530-8ed2-51df7bf91e6c"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
