This is an automated email from the ASF dual-hosted git repository.
blue pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iceberg.git
The following commit(s) were added to refs/heads/master by this push:
new 8a1ebe5 Spac: Add last-partition-id in table metadata (#2224)
8a1ebe5 is described below
commit 8a1ebe534347758ead08aac654af5eb8be1ed71e
Author: jun-he <[email protected]>
AuthorDate: Wed Feb 10 10:55:13 2021 -0800
Spac: Add last-partition-id in table metadata (#2224)
---
site/docs/spec.md | 3 +++
spark2/src/test/java/org/apache/iceberg/examples/README.md | 1 +
2 files changed, 4 insertions(+)
diff --git a/site/docs/spec.md b/site/docs/spec.md
index 97e391a..6bcfd37 100644
--- a/site/docs/spec.md
+++ b/site/docs/spec.md
@@ -498,6 +498,7 @@ Table metadata consists of the following fields:
| _required_ | | **`partition-spec`**| The table’s current
partition spec, stored as only fields. Note that this is used by writers to
partition data, but is not used when reading because reads use the specs stored
in manifest files. (**Deprecated**: use `partition-specs` and
`default-spec-id`instead ) |
| _optional_ | _required_ | **`partition-specs`**| A list of partition specs,
stored as full partition spec objects. |
| _optional_ | _required_ | **`default-spec-id`**| ID of the “current” spec
that writers should use by default. |
+| _optional_ | _required_ | **`last-partition-id`**| An integer; the highest
assigned partition field ID across all partition specs for the table. This is
used to ensure partition fields are always assigned an unused ID when evolving
specs. |
| _optional_ | _optional_ | **`properties`**| A string to string map of table
properties. This is used to control settings that affect reading and writing
and is not intended to be used for arbitrary metadata. For example,
`commit.retry.num-retries` is used to control the number of commit retries. |
| _optional_ | _optional_ | **`current-snapshot-id`**| `long` ID of the
current table snapshot. |
| _optional_ | _optional_ | **`snapshots`**| A list of valid snapshots. Valid
snapshots are snapshots for which all data files exist in the file system. A
data file must not be deleted from the file system until the last snapshot in
which it was listed is garbage collected. |
@@ -894,6 +895,7 @@ Table metadata is serialized as a JSON object according to
the following table.
|**`partition-spec`**|`JSON partition fields (list)`|`See above, read
partition-specs instead`|
|**`partition-specs`**|`JSON partition specs (list of objects)`|`See above`|
|**`default-spec-id`**|`JSON int`|`0`|
+|**`last-partition-id`**|`JSON int`|`1000`|
|**`properties`**|`JSON object: {`<br /> `"<key>": "<val>",`<br
/> `...`<br />`}`|`{`<br /> `"write.format.default":
"avro",`<br /> `"commit.retry.num-retries": "4"`<br />`}`|
|**`current-snapshot-id`**|`JSON long`|`3051729675574597004`|
|**`snapshots`**|`JSON list of objects: [ {`<br /> `"snapshot-id":
<id>,`<br /> `"timestamp-ms": <timestamp-in-ms>,`<br
/> `"summary": {`<br /> `"operation":
<operation>,`<br /> `... },`<br
/> `"manifest-list": "<location>"`<br /> `},`<br
/> `...`<br />`]`|`[ {`<br /> `"snapshot-id":
3051729675574597004,`<br /> `"timestamp-ms": 1515100955770,`<br
/> & [...]
@@ -948,6 +950,7 @@ Writing v2 metadata:
* Table metadata now requires field `table-uuid`.
* Table metadata now requires field `partition-specs`.
* Table metadata now requires field `default-spec-id`.
+* Table metadata now requires field `last-partition-id`.
* Table metadata field `partition-spec` is no longer required and may be
omitted.
* Snapshot added required field field `sequence-number`.
* Snapshot now requires field `manifest-list`.
diff --git a/spark2/src/test/java/org/apache/iceberg/examples/README.md
b/spark2/src/test/java/org/apache/iceberg/examples/README.md
index 4dc9a2a..5f02061 100644
--- a/spark2/src/test/java/org/apache/iceberg/examples/README.md
+++ b/spark2/src/test/java/org/apache/iceberg/examples/README.md
@@ -111,6 +111,7 @@ The metadata file will contain things like the table
location, the schema and th
...
} ]
} ],
+ "last-partition-id" : 1000,
"properties" : { },
"current-snapshot-id" : -1,
"snapshots" : [ ],
