rdblue commented on a change in pull request #3: Store multiple partition specs
in table metadata.
URL: https://github.com/apache/incubator-iceberg/pull/3#discussion_r236341633
##########
File path: core/src/main/java/com/netflix/iceberg/TableMetadata.java
##########
@@ -239,15 +255,42 @@ public Snapshot currentSnapshot() {
public TableMetadata updateTableLocation(String newLocation) {
return new TableMetadata(ops, null, newLocation,
- System.currentTimeMillis(), lastColumnId, schema, spec, properties,
currentSnapshotId,
- snapshots, snapshotLog);
+ System.currentTimeMillis(), lastColumnId, schema, defaultSpecId,
specs, properties,
+ currentSnapshotId, snapshots, snapshotLog);
}
public TableMetadata updateSchema(Schema schema, int lastColumnId) {
+ PartitionSpec.checkCompatibility(spec(), schema);
+ return new TableMetadata(ops, null, location,
+ System.currentTimeMillis(), lastColumnId, schema, defaultSpecId,
specs, properties,
+ currentSnapshotId, snapshots, snapshotLog);
+ }
+
+ public TableMetadata updatePartitionSpec(PartitionSpec spec) {
PartitionSpec.checkCompatibility(spec, schema);
+
+ // if the spec already exists, use the same ID. otherwise, use 1 more than
the highest ID.
+ int newDefaultSpecId = 0;
+ for (Map.Entry<Integer, PartitionSpec> entry : specs.entrySet()) {
+ if (spec.equals(entry.getValue())) {
+ newDefaultSpecId = entry.getKey();
+ break;
+ } else if (newDefaultSpecId <= entry.getKey()) {
+ newDefaultSpecId = entry.getKey() + 1;
+ }
+ }
+
+ Preconditions.checkArgument(defaultSpecId != newDefaultSpecId,
Review comment:
This validates that the spec has changed. It is assumed that if
updatePartitionSpec is called, the intent was to change the spec. In
buildReplacement, the entire table state will be replaced. The new table can
have the same partition spec as the old, or it can use the same one. There
isn't a requirement to change the spec.
There may be a situation where a user attempts to update the partition spec
without actually changing it, but I thought that this should be conservative
and validate. If the spec doesn't actually change because a user attempted to
update the spec to the current one, then some other component should catch it
and avoid committing a change entirely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
