[GitHub] [hudi] boneanxs commented on a diff in pull request #8076: [HUDI-5884] Support bulk_insert for insert_overwrite and insert_overwrite_table

Mon, 22 May 2023 23:52:26 -0700 


boneanxs commented on code in PR #8076:
URL: https://github.com/apache/hudi/pull/8076#discussion_r1201621093


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkDataSourceDAGExecution.scala:
##########
@@ -89,7 +89,7 @@ class TestSparkDataSourceDAGExecution extends 
HoodieSparkClientTestBase with Sca
   @CsvSource(Array(
     "upsert,org.apache.hudi.client.SparkRDDWriteClient.commit",
     "insert,org.apache.hudi.client.SparkRDDWriteClient.commit",
-    "bulk_insert,org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow"))
+    "bulk_insert,org.apache.hudi.HoodieDatasetBulkInsertHelper.bulkInsert"))

Review Comment:
   `bulk_insert` with `Overwrite` mode will change to use 
`DatasetBulkInsertOverwriteTableCommitActionExecutor` to avoid deleting 
basePath first. So it would call `FSUtils.getAllPartitionPaths` to get all 
existing partition paths, which would introduce extra stages containing 
`HoodieSparkSqlWriter$.bulkInsertAsRow`, but it won't call write operation 
multi times, can see from the screenshot.
   
   ![Screenshot 2023-05-23 at 14 45 
58](https://github.com/apache/hudi/assets/10115332/d2e0cf3b-65ed-45a3-b786-0778f9b8e395)
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to