yihua commented on code in PR #7476:
URL: https://github.com/apache/hudi/pull/7476#discussion_r1060867754
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2449,12 +2450,17 @@ public Builder withWriteBufferLimitBytes(int
writeBufferLimit) {
}
public Builder withWriteWaitStrategy(String waitStrategy) {
- writeConfig.setValue(WRITE_WAIT_STRATEGY, String.valueOf(waitStrategy));
+ writeConfig.setValue(WRITE_EXECUTOR_DISRUPTOR_WAIT_STRATEGY,
String.valueOf(waitStrategy));
return this;
}
public Builder withWriteBufferSize(int size) {
- writeConfig.setValue(WRITE_DISRUPTOR_BUFFER_SIZE, String.valueOf(size));
+ writeConfig.setValue(WRITE_EXECUTOR_DISRUPTOR_BUFFER_SIZE,
String.valueOf(size));
+ return this;
+ }
Review Comment:
nit: If this is not intended to use as a public API, could you remove this
and only keep `Builder#withWriteExecutorDisruptorWriteBufferSize` which has the
same functionality?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2857,8 +2863,15 @@ private void validate() {
}
public HoodieWriteConfig build() {
+ return build(true);
+ }
+
+ @VisibleForTesting
+ public HoodieWriteConfig build(boolean shouldValidate) {
Review Comment:
should this method use the default scope/visibility instead of `public`?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestSimpleExecutionInSpark.java:
##########
@@ -88,10 +81,10 @@ public Integer finish() {
return count;
}
};
- SimpleHoodieExecutor<HoodieRecord, Tuple2<HoodieRecord,
Option<IndexedRecord>>, Integer> exec = null;
+ SimpleExecutor<HoodieRecord, Tuple2<HoodieRecord, Option<IndexedRecord>>,
Integer> exec = null;
try {
- exec = new SimpleHoodieExecutor(hoodieRecords.iterator(), consumer,
getCloningTransformer(HoodieTestDataGenerator.AVRO_SCHEMA));
+ exec = new SimpleExecutor(hoodieRecords.iterator(), consumer,
Function.identity());
Review Comment:
Still use `getTransformer` instead of `Function.identity()` for the tests?
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/execution/TestDisruptorExecutionInSpark.java:
##########
@@ -35,24 +36,27 @@
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Timeout;
+import scala.Tuple2;
import java.util.ArrayList;
import java.util.List;
-import scala.Tuple2;
-
-import static
org.apache.hudi.execution.HoodieLazyInsertIterable.getCloningTransformer;
+import static
org.apache.hudi.execution.HoodieLazyInsertIterable.getTransformer;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.when;
public class TestDisruptorExecutionInSpark extends HoodieClientTestHarness {
private final String instantTime =
HoodieActiveTimeline.createNewInstantTime();
+
+ private final HoodieWriteConfig writeConfig = HoodieWriteConfig.newBuilder()
Review Comment:
nit: (not required in this PR) the tests using different types of write
executors should be generalized in a base class, like `TestWriteMarkersBase`,
instead of duplicating the code. This can be refactored later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
