szehon-ho commented on code in PR #3482:
URL: https://github.com/apache/iceberg/pull/3482#discussion_r844476218
##########
docs/spark/spark-queries.md:
##########
@@ -229,53 +224,106 @@ join prod.db.table.snapshots s
on h.snapshot_id = s.snapshot_id
order by made_current_at
```
-```text
-+-------------------------+-----------+----------------+---------------------+----------------------------------+
-| made_current_at | operation | snapshot_id | is_current_ancestor |
summary[spark.app.id] |
-+-------------------------+-----------+----------------+---------------------+----------------------------------+
+
+| made_current_at | operation | snapshot_id | is_current_ancestor |
summary[spark.app.id] |
+| -- | -- | -- | -- | -- |
| 2019-02-08 03:29:51.215 | append | 57897183625154 | true |
application_1520379288616_155055 |
| 2019-02-09 16:24:30.13 | delete | 29641004024753 | false |
application_1520379288616_151109 |
| 2019-02-09 16:32:47.336 | append | 57897183625154 | true |
application_1520379288616_155055 |
| 2019-02-08 03:47:55.948 | overwrite | 51792995261850 | true |
application_1520379288616_152431 |
-+-------------------------+-----------+----------------+---------------------+----------------------------------+
-```
### Files
-To show a table's data files and each file's metadata, run:
+To show a table's current data files:
```sql
SELECT * FROM prod.db.table.files
```
-| content | file_path | file_format | spec_id | partition | record_count |
file_size_in_bytes | column_sizes | value_counts | null_value_counts |
nan_value_counts | lower_bounds | upper_bounds | key_metadata | split_offsets |
equality_ids | sort_order_id |
+
+|content|file_path
|file_format|spec_id|partition|record_count|file_size_in_bytes|column_sizes
|value_counts |null_value_counts|nan_value_counts|lower_bounds
|upper_bounds |key_metadata|split_offsets|equality_ids|sort_order_id|
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
-- | -- |
| 0 | s3:/.../table/data/00000-3-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet
| PARQUET | 0 | {1999-01-01, 01} | 1 | 597 | [1 ->
90, 2 -> 62] | [1 -> 1, 2 -> 1] | [1 -> 0, 2 -> 0] | [] | [1 ->
, 2 -> c] | [1 -> , 2 -> c] | null | [4] | null | null |
| 0 | s3:/.../table/data/00001-4-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet
| PARQUET | 0 | {1999-01-01, 02} | 1 | 597 | [1 ->
90, 2 -> 62] | [1 -> 1, 2 -> 1] | [1 -> 0, 2 -> 0] | [] | [1 ->
, 2 -> b] | [1 -> , 2 -> b] | null | [4] | null | null |
| 0 | s3:/.../table/data/00002-5-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet
| PARQUET | 0 | {1999-01-01, 03} | 1 | 597 | [1 ->
90, 2 -> 62] | [1 -> 1, 2 -> 1] | [1 -> 0, 2 -> 0] | [] | [1 ->
, 2 -> a] | [1 -> , 2 -> a] | null | [4] | null | null |
### Manifests
-To show a table's file manifests and each file's metadata, run:
+To show a table's current file manifests:
```sql
SELECT * FROM prod.db.table.manifests
```
-```text
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
-| path |
length | partition_spec_id | added_snapshot_id | added_data_files_count |
existing_data_files_count | deleted_data_files_count | partition_summaries
|
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
+
+| path | length | partition_spec_id | added_snapshot_id |
added_data_files_count | existing_data_files_count | deleted_data_files_count |
partition_summaries |
+| -- | -- | -- | -- | -- | -- | -- | -- |
| s3://.../table/metadata/45b5290b-ee61-4788-b324-b1e2735c0e10-m0.avro | 4479
| 0 | 6668963634911763636 | 8 | 0
| 0 |
[[false,null,2019-05-13,2019-05-15]] |
-+----------------------------------------------------------------------+--------+-------------------+---------------------+------------------------+---------------------------+--------------------------+--------------------------------------+
+
+Note:
+1. Fields within `partition_summaries` column of the manifests table
correspond to `field_summary` structs within [manifest
list](../../../spec#manifest-lists), with the following order:
+ - `contains_null`
+ - `contains_nan`
+ - `lower_bound`
+ - `upper_bound`
+2. `contains_nan` could return null, which indicates that this information is
not available from the file's metadata.
+ This usually occurs when reading from V1 table, where `contains_nan` is
not populated.
+
+### Partitions
+
+To show a table's current partitions:
+
+```sql
+SELECT * FROM prod.db.table.partitions
```
-Note:
-1. Fields within `partition_summaries` column of the manifests table
correspond to `field_summary` structs within [manifest
list](../../../spec#manifest-lists), with the following order:
- - `contains_null`
- - `contains_nan`
- - `lower_bound`
- - `upper_bound`
-2. `contains_nan` could return null, which indicates that this information is
not available from files' metadata.
- This usually occurs when reading from V1 table, where `contains_nan` is not
populated.
+| partition | record_count | file_count |
+| -- | -- | -- |
+| {20211001, 11}| 1| 1|
+| {20211002, 11}| 1| 1|
+| {20211001, 10}| 1| 1|
+| {20211002, 10}| 1| 1|
+
+### All Metadata Tables
+
+These tables are unions of the metadata tables specific to the current
snapshot, and return metadata across all snapshots.
+
+{{< hint danger >}}
+The "all" metadata tables may produce more than one row per data file or
manifest file because metadata files may be part of more than one table
snapshot.
+{{< /hint >}}
+
+#### All Data Files
+
+To show all the table's data files and each file's metadata:
+
+```sql
+SELECT * FROM prod.db.table.all_data_files
+```
+
+| content | file_path | file_format | partition | record_count |
file_size_in_bytes | column_sizes| value_counts | null_value_counts |
nan_value_counts| lower_bounds|
upper_bounds|key_metadata|split_offsets|equality_ids|sort_order_id|
+| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
-- |
+| 0|s3:.../dt=20210102/xxx.parquet| PARQUET|{20210102}| 14|
2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}|
{}|{1 -> 1, 2 -> 20210102}|{1 -> 2, 2 -> 20210102}| null|
[4]| null| 0|
+| 0|s3:.../dt=20210103/xxx.parquet| PARQUET|{20210103}| 14|
2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}|
{}|{1 -> 1, 2 -> 20210103}|{1 -> 3, 2 -> 20210103}| null|
[4]| null| 0|
+| 0|sA valid manifest file is one that is3:.../dt=20210104/xxx.parquet|
PARQUET|{20210104}| 14| 2444|{1 -> 94, 2 -> 17}|{1 ->
14, 2 -> 14}| {1 -> 0, 2 -> 0}| {}|{1 -> 1, 2 -> 20210104}|{1 ->
3, 2 -> 20210104}| null| [4]| null| 0|
+
+#### All Manifests
+
+To show all the table's manifest files:
+
+```sql
+SELECT * FROM prod.db.table.all_manifests
+```
+
+| path | length | partition_spec_id | added_snapshot_id |
added_data_files_count | existing_data_files_count | deleted_data_files_count|
partition_summaries|
+| -- | -- | -- | -- | -- | -- | -- | -- |
+| s3:/.../table/data/dt=20210102/xxx.parquet | 6376 | 0 | 6272782676904868561
| 2 | 0 | 0 |[{false, false, 10, 11}, {false, false, 20210101, 20210101}]|
Review Comment:
No problem, these are just small points.
I mean, in the other tables examples they show a realistic path, for
example, the files table shows:
s3:/.../table/data/00000-3-8d6d60e8-d427-4809-bcf0-f5d45a4aad96.parquet
My point was just, its a bit inconsistent in the doc to suddenly jump from
the example using real possible parquet file names, to a fake ones like
xxx.parquet.
For the second point, the second partition value is {false, false, 20210101,
20210101} , looks like date, but the first one is {false, false, 10, 11}, which
does not look like date. So just wondering if we can make it more consistent
and have both be dates or something. Again, not a huge point, but just feels
like a more realistic example for users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
