openinx commented on a change in pull request #2499:
URL: https://github.com/apache/iceberg/pull/2499#discussion_r619946868
##########
File path:
spark/src/main/java/org/apache/iceberg/spark/source/SparkAppenderFactory.java
##########
@@ -85,6 +72,52 @@
this.posDeleteRowSchema = posDeleteRowSchema;
}
+ public static SparkAppenderFactoryBuilder builderFor(Table table, Schema
writeSchema, StructType dsSchema) {
+ return new SparkAppenderFactoryBuilder(table, writeSchema, dsSchema);
+ }
+
+ public static class SparkAppenderFactoryBuilder {
+ private final Map<String, String> properties;
+ private final Schema writeSchema;
+ private final StructType dsSchema;
+ private PartitionSpec spec;
+ private int[] equalityFieldIds;
+ private Schema eqDeleteRowSchema;
+ private Schema posDeleteRowSchema;
+
+ SparkAppenderFactoryBuilder(Table table, Schema writeSchema, StructType
dsSchema) {
+ this.properties = table.properties();
+ this.spec = table.spec();
+ this.writeSchema = writeSchema;
+ this.dsSchema = dsSchema;
+ }
+
+ public SparkAppenderFactoryBuilder spec(PartitionSpec newSpec) {
+ this.spec = newSpec;
+ return this;
+ }
+
+ public SparkAppenderFactoryBuilder equalityFieldIds(int[]
newEqualityFieldIds) {
+ this.equalityFieldIds = newEqualityFieldIds;
+ return this;
+ }
+
+ public SparkAppenderFactoryBuilder eqDeleteRowSchema(Schema
newEqDeleteRowSchema) {
+ this.eqDeleteRowSchema = newEqDeleteRowSchema;
+ return this;
+ }
+
+ public SparkAppenderFactoryBuilder posDelRowSchema(Schema
newPosDelRowSchema) {
+ this.posDeleteRowSchema = newPosDelRowSchema;
+ return this;
+ }
+
+ public SparkAppenderFactory build() {
+ return new SparkAppenderFactory(properties, writeSchema, dsSchema, spec,
equalityFieldIds,
Review comment:
We will need to do the necessary validation before we construct the
`SparkAppenderFactory` instance:
1. The `properties`, `writeSchema`, `dsSchema` must not be null;
2. If the `equalityFieldIds` is not null, means people plan to write
equality deletes into delete files. Then the `eqDeleteRowSchema` must not be
null; Another side, if `eqDeleteRowSchema` is not null, then
`equalityFieldIds` must not be null.
3. the `posDeleteRowSchema` can always choose to be null or not.
##########
File path:
spark/src/main/java/org/apache/iceberg/spark/source/SparkAppenderFactory.java
##########
@@ -85,6 +72,52 @@
this.posDeleteRowSchema = posDeleteRowSchema;
}
+ public static SparkAppenderFactoryBuilder builderFor(Table table, Schema
writeSchema, StructType dsSchema) {
+ return new SparkAppenderFactoryBuilder(table, writeSchema, dsSchema);
+ }
+
+ public static class SparkAppenderFactoryBuilder {
Review comment:
Since the class `SparkAppenderFactory` was not marked as `public`, so I
think the newly introduced `SparkAppenderFactoryBuilder` also don't have to be
`public` ?
btw, I think we could just use the name `Builder` because it's in the
class `SparkAppenderFactory`, the prefix `SparkAppenderFactory` seems to be
redundant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
