KnightChess commented on code in PR #3482:
URL: https://github.com/apache/iceberg/pull/3482#discussion_r843411054
##########
docs/spark/spark-queries.md:
##########
@@ -229,53 +224,106 @@ join prod.db.table.snapshots s
on h.snapshot_id = s.snapshot_id
order by made_current_at
```
-```text
-+-------------------------+-----------+----------------+---------------------+----------------------------------+
-| made_current_at | operation | snapshot_id | is_current_ancestor |
summary[spark.app.id] |
-+-------------------------+-----------+----------------+---------------------+----------------------------------+
+
+| made_current_at | operation | snapshot_id | is_current_ancestor |
summary[spark.app.id] |
+| -- | -- | -- | -- | -- |
| 2019-02-08 03:29:51.215 | append | 57897183625154 | true |
application_1520379288616_155055 |
| 2019-02-09 16:24:30.13 | delete | 29641004024753 | false |
application_1520379288616_151109 |
| 2019-02-09 16:32:47.336 | append | 57897183625154 | true |
application_1520379288616_155055 |
| 2019-02-08 03:47:55.948 | overwrite | 51792995261850 | true |
application_1520379288616_152431 |
-+-------------------------+-----------+----------------+---------------------+----------------------------------+
-```
### Files
-To show a table's data files and each file's metadata, run:
+To show a table's current data files:
```sql
SELECT * FROM prod.db.table.files
```
-| content | file_path | file_format | spec_id | partition | record_count |
file_size_in_bytes | column_sizes | value_counts | null_value_counts |
nan_value_counts | lower_bounds | upper_bounds | key_metadata | split_offsets |
equality_ids | sort_order_id |
+
+|content|file_path
|file_format|spec_id|partition|record_count|file_size_in_bytes|column_sizes
|value_counts |null_value_counts|nan_value_counts|lower_bounds
|upper_bounds |key_metadata|split_offsets|equality_ids|sort_order_id|
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
-- | -- |
| 0 | s3:/.../table/data/00000-3-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet
| PARQUET | 0 | {1999-01-01, 01} | 1 | 597 | [1 ->
90, 2 -> 62] | [1 -> 1, 2 -> 1] | [1 -> 0, 2 -> 0] | [] | [1 ->
, 2 -> c] | [1 -> , 2 -> c] | null | [4] | null | null |
| 0 | s3:/.../table/data/00001-4-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet
| PARQUET | 0 | {1999-01-01, 02} | 1 | 597 | [1 ->
90, 2 -> 62] | [1 -> 1, 2 -> 1] | [1 -> 0, 2 -> 0] | [] | [1 ->
, 2 -> b] | [1 -> , 2 -> b] | null | [4] | null | null |
| 0 | s3:/.../table/data/00002-5-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet
| PARQUET | 0 | {1999-01-01, 03} | 1 | 597 | [1 ->
90, 2 -> 62] | [1 -> 1, 2 -> 1] | [1 -> 0, 2 -> 0] | [] | [1 ->
, 2 -> a] | [1 -> , 2 -> a] | null | [4] | null | null |
### Manifests
-To show a table's file manifests and each file's metadata, run:
+To show a table's current file manifests:
```sql
SELECT * FROM prod.db.table.manifests
```
-```text
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
-| path |
length | partition_spec_id | added_snapshot_id | added_data_files_count |
existing_data_files_count | deleted_data_files_count | partition_summaries
|
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
+
+| path | length | partition_spec_id | added_snapshot_id |
added_data_files_count | existing_data_files_count | deleted_data_files_count |
partition_summaries |
+| -- | -- | -- | -- | -- | -- | -- | -- |
| s3://.../table/metadata/45b5290b-ee61-4788-b324-b1e2735c0e10-m0.avro | 4479
| 0 | 6668963634911763636 | 8 | 0
| 0 |
[[false,null,2019-05-13,2019-05-15]] |
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
+
+Note:
+1. Fields within `partition_summaries` column of the manifests table
correspond to `field_summary` structs within [manifest
list](../../../spec#manifest-lists), with the following order:
+ - `contains_null`
+ - `contains_nan`
+ - `lower_bound`
+ - `upper_bound`
+2. `contains_nan` could return null, which indicates that this information is
not available from the file's metadata.
+ This usually occurs when reading from V1 table, where `contains_nan` is
not populated.
+
+### Partitions
+
+To show a table's current partitions:
+
+```sql
+SELECT * FROM prod.db.table.partitions
```
-Note:
-1. Fields within `partition_summaries` column of the manifests table
correspond to `field_summary` structs within [manifest
list](../../../spec#manifest-lists), with the following order:
- - `contains_null`
- - `contains_nan`
- - `lower_bound`
- - `upper_bound`
-2. `contains_nan` could return null, which indicates that this information is
not available from files' metadata.
- This usually occurs when reading from V1 table, where `contains_nan` is not
populated.
+| partition | record_count | file_count |
+| -- | -- | -- |
+| {20211001, 11}| 1| 1|
+| {20211002, 11}| 1| 1|
+| {20211001, 10}| 1| 1|
+| {20211002, 10}| 1| 1|
+
+### All Metadata Tables
+
+These tables are unions of the metadata tables specific to the current
snapshot, and return metadata across all snapshots.
+
+{{< hint danger >}}
+The "all" metadata tables may produce more than one row per data file or
manifest file because metadata files may be part of more than one table
snapshot.
+{{< /hint >}}
+
+#### All Data Files
+
+To show all the table's data files and each file's metadata:
+
+```sql
+SELECT * FROM prod.db.table.all_data_files
+```
+
+| content | file_path | file_format | partition | record_count |
file_size_in_bytes | column_sizes| value_counts | null_value_counts |
nan_value_counts| lower_bounds|
upper_bounds|key_metadata|split_offsets|equality_ids|sort_order_id|
+| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
-- |
+| 0|s3:.../dt=20210102/xxx.parquet| PARQUET|{20210102}| 14|
2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}|
{}|{1 -> 1, 2 -> 20210102}|{1 -> 2, 2 -> 20210102}| null|
[4]| null| 0|
+| 0|s3:.../dt=20210103/xxx.parquet| PARQUET|{20210103}| 14|
2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}|
{}|{1 -> 1, 2 -> 20210103}|{1 -> 3, 2 -> 20210103}| null|
[4]| null| 0|
+| 0|sA valid manifest file is one that is3:.../dt=20210104/xxx.parquet|
PARQUET|{20210104}| 14| 2444|{1 -> 94, 2 -> 17}|{1 ->
14, 2 -> 14}| {1 -> 0, 2 -> 0}| {}|{1 -> 1, 2 -> 20210104}|{1 ->
3, 2 -> 20210104}| null| [4]| null| 0|
+
+#### All Manifests
+
+To show all the table's manifest files:
+
+```sql
+SELECT * FROM prod.db.table.all_manifests
+```
+
+| path | length | partition_spec_id | added_snapshot_id |
added_data_files_count | existing_data_files_count | deleted_data_files_count|
partition_summaries|
+| -- | -- | -- | -- | -- | -- | -- | -- |
+| s3:/.../table/data/dt=20210102/xxx.parquet | 6376 | 0 | 6272782676904868561
| 2 | 0 | 0 |[{false, false, 10, 11}, {false, false, 20210101, 20210101}]|
Review Comment:
Sorry, I don't quite understand this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
