rdblue commented on a change in pull request #23606: [SPARK-26666][SQL] Support DSv2 overwrite and dynamic partition overwrite. URL: https://github.com/apache/spark/pull/23606#discussion_r250025352
########## File path: sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/SupportsOverwrite.java ########## @@ -0,0 +1,36 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.sources.v2.writer; + +import org.apache.spark.sql.sources.Filter; + +/** + * Write builder trait for tables that support overwrite by filter. + * <p> + * Overwriting data by filter will delete any data that matches the filter and replace it with data + * that is committed in the write. + */ +public interface SupportsOverwrite extends WriteBuilder { Review comment: This is needed for a couple use cases: 1. `INSERT OVERWRITE ... PARTITION ...` with static partitions in static overwrite mode, for compatibility with existing queries. 2. Situations where users would currently call `INSERT OVERWRITE`, but should be explicit. This operation is a combined delete and replace, [ReplaceData in the SPIP](https://docs.google.com/document/d/1gYm5Ji2Mge3QBdOliFV5gSPTKlX4q1DCBXIkiyMv62A/edit?ts=5a987801#heading=h.vgt28bdlilsp). This is currently implemented by the existing `INSERT OVERWRITE` statement. `INSERT OVERWRITE` (as defined by Hive) is dynamic so what gets deleted is implicit. Implicit deletes lead to confusion, so it would be better to move users to patterns that explicitly replace data. An example of such a pattern is an hourly job that is idempotent and produces a summary for a day. Each hour, it overwrites the summary from the last hour, until the day is finished and the last summary is left. We are moving users to structure this query to explicitly overwrite using an expression, which we can also use to warn if the user writes data that wouldn't be replaced by running the query again. A second use for this is static overwrite mode, which is a Spark-only behavior. In static mode, Spark will drop any matching static partitions and then insert data. In that case, we can't use dynamic overwrite because it wouldn't delete all of the static partitions. Instead, we translate the static partitions to an overwrite expression. So using this, we can implement all overwrite behaviors using the v2 API. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
