aokolnychyi commented on code in PR #6651:
URL: https://github.com/apache/iceberg/pull/6651#discussion_r1115022514
##########
core/src/main/java/org/apache/iceberg/util/SnapshotUtil.java:
##########
@@ -394,4 +395,19 @@ public static Schema schemaFor(Table table, Long
snapshotId, Long timestampMilli
return table.schema();
}
+
+ /**
+ * Fetch the snapshot at the head of the given branch in the given table
Review Comment:
nit: Missing `.` at the end of the sentence?
##########
core/src/main/java/org/apache/iceberg/util/SnapshotUtil.java:
##########
@@ -394,4 +395,19 @@ public static Schema schemaFor(Table table, Long
snapshotId, Long timestampMilli
return table.schema();
}
+
+ /**
+ * Fetch the snapshot at the head of the given branch in the given table
+ *
+ * @param table a {@link Table}
+ * @param branch branch
+ * @return Snapshot
Review Comment:
What about adding more details?
```
@return the latest snapshot for the given branch
```
##########
core/src/main/java/org/apache/iceberg/util/SnapshotUtil.java:
##########
@@ -394,4 +395,19 @@ public static Schema schemaFor(Table table, Long
snapshotId, Long timestampMilli
return table.schema();
}
+
+ /**
+ * Fetch the snapshot at the head of the given branch in the given table
+ *
+ * @param table a {@link Table}
+ * @param branch branch
Review Comment:
nit: `a branch`?
##########
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestDataFrameWriterV2.java:
##########
@@ -193,6 +194,46 @@ public void testMergeSchemaIcebergProperty() throws
Exception {
sql("select * from %s order by id", tableName));
}
+ @Test
+ public void testMergeSchemaOnBranch() throws Exception {
+ String branch = "test-branch";
+
+ sql(
+ "ALTER TABLE %s SET TBLPROPERTIES ('%s'='true')",
+ tableName, TableProperties.SPARK_WRITE_ACCEPT_ANY_SCHEMA);
+
+ Dataset<Row> twoColDF =
+ jsonToDF(
+ "id bigint, data string",
+ "{ \"id\": 1, \"data\": \"a\" }",
+ "{ \"id\": 2, \"data\": \"b\" }");
+
+ twoColDF.writeTo(tableName).option(SparkWriteOptions.BRANCH,
branch).append();
+
+ assertEquals(
+ "Should have initial 2-column rows",
+ ImmutableList.of(row(1L, "a"), row(2L, "b")),
+ sql("select * from %s version as of '%s' order by id", tableName,
branch));
+
+ Dataset<Row> threeColDF =
+ jsonToDF(
+ "id bigint, data string, new_col float",
+ "{ \"id\": 3, \"data\": \"c\", \"new_col\": 12.06 }",
+ "{ \"id\": 4, \"data\": \"d\", \"new_col\": 14.41 }");
+
+ threeColDF
+ .writeTo(tableName)
+ .option("merge-schema", "true")
Review Comment:
nit: Shall we use `SparkWriteOptions.MERGE_SCHEMA`?
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java:
##########
@@ -324,4 +325,12 @@ public boolean caseSensitive() {
.defaultValue(SQLConf.CASE_SENSITIVE().defaultValueString())
.parse();
}
+
+ public String branch() {
+ return confParser
+ .stringConf()
+ .option(SparkWriteOptions.BRANCH)
+ .defaultValue(SnapshotRef.MAIN_BRANCH)
Review Comment:
Makes sense to me.
##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkPositionDeltaWriteBuilder.java:
##########
@@ -81,6 +82,11 @@ public DeltaWrite build() {
handleTimestampWithoutZone ||
!SparkUtil.hasTimestampWithoutZone(table.schema()),
SparkUtil.TIMESTAMP_WITHOUT_TIMEZONE_ERROR);
+ if (!writeConf.branch().equalsIgnoreCase(SnapshotRef.MAIN_BRANCH)) {
+ throw new UnsupportedOperationException(
+ "Row level operations on non-main branches are currently
unsupported");
+ }
Review Comment:
I don't think we need a separate test for this. There is no way to set it
anyway. If we want to have this validation, what about negating the sentence?
```
throw new UnsupportedOperationException(
"Row-level operations are currently supported only on the main branch");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
