RussellSpitzer commented on pull request #3459:
URL: https://github.com/apache/iceberg/pull/3459#issuecomment-970426229
@knightchess here is an example of what happens with the partitions metadata
table
```
scala> spark.sql("CREATE TABLE example (a int, b int, c int, d int) using
Iceberg location 'file:///Users/russellspitzer/Temp/example'")
scala> spark.sql("ALTER TABLE example ADD PARTITION FIELD identity(a)")
scala> spark.sql("INSERT INTO example VALUES (1,1,1,1)")
// Partition (a = 1)
scala> spark.sql("ALTER TABLE example ADD PARTITION FIELD identity(b)")
scala> spark.sql("INSERT INTO example VALUES (2,2,2,2)")
// Partition (a = 2, b =2)
scala> spark.sql("ALTER TABLE example ADD PARTITION FIELD identity(c)")
scala> spark.sql("INSERT INTO example VALUES (3,3,3,3)")
// Partition (a = 3, b = 3, c = 3)
scala> spark.sql("ALTER TABLE example DROP PARTITION FIELD identity(c)")
scala> spark.sql("INSERT INTO example VALUES (4,4,4,4)")
// Partition (a = 4, b = 4, null(c) = null)
scala> spark.sql("ALTER TABLE example DROP PARTITION FIELD identity(b)")
scala> spark.sql("INSERT INTO example VALUES (5,5,5,5)")
// Partition (a= 5, null(b) = null, null(c) = null)
scala> spark.sql("ALTER TABLE example DROP PARTITION FIELD identity(a)")
scala> spark.sql("INSERT INTO example VALUES (6,6,6,6)")
// Partition (null(a) = null, null(b) = null, null(c) = null
```
This creates a metadata table
```
scala> spark.sql("SELECT * from default.example.partitions SORT BY
partition").show
+------------------+------------+----------+
| partition|record_count|file_count|
+------------------+------------+----------+
|{null, null, null}| 1| 1|
| {1, null, null}| 1| 1|
| {2, 2, null}| 1| 1|
| {3, 3, 3}| 1| 1|
| {4, 4, null}| 1| 1|
| {5, null, null}| 1| 1|
+------------------+------------+----------+
```
This metadata table has a partition spec which covers the entire range of
possible specs the table has had even though the underlying table is currently
unpartitioned. Now even though some partitions where written in a different
spec (in this case every partition was written with a different spec) you see
them all in the context of the "maximal" spec which covers all fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
