nsivabalan commented on a change in pull request #3149:
URL: https://github.com/apache/hudi/pull/3149#discussion_r664603586
##########
File path:
hudi-spark-datasource/hudi-spark-common/src/main/java/org/apache/hudi/internal/BulkInsertDataInternalWriterHelper.java
##########
@@ -92,19 +95,30 @@ public void write(InternalRow record) throws IOException {
public void abort() {
}
- private void createNewHandle(String partitionPath) throws IOException {
- if (null != handle) {
- close();
+ private HoodieRowCreateHandle getRowCreateHandle(String partitionPath)
throws IOException {
+ if (!handles.containsKey(partitionPath)) { // if there is no handle
corresponding to the partition path
+ // if records are sorted, we can close all existing handles
+ if (writeConfig.getBulkInsertIsPartitionRecordsSorted()) {
Review comment:
here is the usage of arcPartitionRecordsSorted. basically, whether to
cache and reuse the write handles or close it right away when switching to a
diff partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
