rdblue commented on a change in pull request #922: URL: https://github.com/apache/iceberg/pull/922#discussion_r449921994
########## File path: core/src/main/java/org/apache/iceberg/PartitionSpecUpdate.java ########## @@ -0,0 +1,284 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iceberg; + +import java.util.Arrays; +import java.util.Comparator; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Consumer; +import java.util.function.Function; +import java.util.stream.Collectors; +import org.apache.iceberg.exceptions.ValidationException; +import org.apache.iceberg.relocated.com.google.common.annotations.VisibleForTesting; +import org.apache.iceberg.relocated.com.google.common.base.Preconditions; +import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList; +import org.apache.iceberg.relocated.com.google.common.collect.Lists; +import org.apache.iceberg.relocated.com.google.common.collect.Maps; +import org.apache.iceberg.transforms.Transforms; + +/** + * PartitionSpec evolution API implementation. + */ +class PartitionSpecUpdate implements UpdatePartitionSpec { + + private static final String SOFT_DELETE_POSTFIX = "__[removed]"; + + private final TableMetadata base; + private final TableOperations ops; + private final List<PartitionSpec> specs; + private final Schema schema; + private final Map<String, PartitionField> curSpecFields; + private final List<Consumer<PartitionSpec.Builder>> newSpecFields = Lists.newArrayList(); + private final Map<String, PartitionField> newRemovedFields = Maps.newHashMap(); Review comment: I think the way that this operation keeps track of state makes it very confusing to read and understand. I can see the appeal of trying to just delegating to the spec builder, but the current approach requires complicated checks after the builder is configured (like `checkIfRemoved` on line 97) and methods that need to rewrite the entire spec anyway (like `freshSpecFieldIds` and `fillGapsByNonNullFields`). I think that this would be easier to understand if it did validation and assignment incrementally. Instead of adding a method to rewrite the spec with different field IDs, I think this should check for deleted fields when the configuration method is called and use the correct ID from the start. This should help avoid the need for work-arounds like `getLastPartitionField` in the spec builder. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
