jackylk commented on a change in pull request #3589: [CARBONDATA-3667] Insert
stage recover processing of the partition ta…
URL: https://github.com/apache/carbondata/pull/3589#discussion_r369389522
##########
File path:
integration/spark2/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertFromStageCommand.scala
##########
@@ -431,8 +318,53 @@ case class CarbonInsertFromStageCommand(
}.map { future =>
future.get()
}
+ }
+
+ /**
+ * Delete stage file and success file with retry
+ */
+ private def deleteStageFilesWithRetry(
+ executorService: ExecutorService,
+ _stageFiles: Array[(CarbonFile, CarbonFile)]): Unit = {
+ val startTime = System.currentTimeMillis()
+ var retryCount = DELETE_STAGEFILES_RETRY_TIMES
+ var stageFiles = _stageFiles
+ while (stageFiles.length > 0 && retryCount > 0) {
+ retryCount -= 1
+ // 1) delete stage files
+ deleteStageFiles(executorService, stageFiles)
+ // 2) ensure stage files are all deleted
+ stageFiles = ensureStageFilesAreAllDeleted(executorService, stageFiles)
Review comment:
I think this will slow down the performance, can we check it by using the
return value of delete operation in `deleteStageFiles`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
