giucris commented on code in PR #7607: URL: https://github.com/apache/iceberg/pull/7607#discussion_r1334958313
########## spark/v3.4/spark-extensions/src/main/scala/org/apache/iceberg/spark/extensions/IcebergMergeInto.scala: ########## @@ -0,0 +1,399 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.iceberg.spark.extensions + +import org.apache.spark.sql.Column +import org.apache.spark.sql.Dataset +import org.apache.spark.sql.Row +import org.apache.spark.sql.SparkSession +import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation +import org.apache.spark.sql.catalyst.expressions.Expression +import org.apache.spark.sql.catalyst.plans.logical.Assignment +import org.apache.spark.sql.catalyst.plans.logical.DeleteAction +import org.apache.spark.sql.catalyst.plans.logical.InsertAction +import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction +import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan +import org.apache.spark.sql.catalyst.plans.logical.MergeAction +import org.apache.spark.sql.catalyst.plans.logical.MergeIntoContext +import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable +import org.apache.spark.sql.catalyst.plans.logical.UpdateAction +import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction +import org.apache.spark.sql.functions.expr +import scala.collection.JavaConverters + + +object IcebergMergeInto { Review Comment: I did some research and I need more details on how you would like the merge api options to work. 1. Do we want to set the options at the individual action level or at the global merge into level or both? 2. What kind of options do we want to support? Runtime write config or Table properties or both? To date and following the iceberg docs, the options really useful (IMHO) for merge, e.g., write.merge.mode, write.delete.mode, etc., are table properties and are currently set on the target table side during create or after altering the table. However, there are some runtime configurations that could be provided at write time, [these](https://iceberg.apache.org/docs/latest/spark-configuration/#write-options) 3. In case we also want to support table properties, what is expected after the merge operation with configured table properties? Target table containing the new properties or we want that the properties are used only in that runtime (In this case I don't know how feasible it is, I need to investigate further) Given all these open points, can we addressing this need as a new feature, thus in a subsequent PR of the IcebergMergeInto API? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
