marchpure commented on a change in pull request #3799:
URL: https://github.com/apache/carbondata/pull/3799#discussion_r443451334



##########
File path: 
integration/spark/src/main/scala/org/apache/spark/sql/execution/command/management/CarbonInsertFromStageCommand.scala
##########
@@ -148,10 +149,19 @@ case class CarbonInsertFromStageCommand(
         return Seq.empty
       }
 
-      // 2) read all stage files to collect input files for data loading
-      // create a thread pool to read them
+      // We add a tag 'loading' to the stages in process.
+      // different insertstage processes can load different data separately
+      // by choose the stages without 'loading' tag or stages loaded timeout.
+      // which avoid loading the same data between concurrent insertstage 
processes.
+      // The 'loading' tag is actually an empty file with
+      // '.loading' suffix filename
       val numThreads = Math.min(Math.max(stageFiles.length, 1), 10)
       val executorService = Executors.newFixedThreadPool(numThreads)
+      createStageLoadingFilesWithRetry(executorService, stageFiles)
+      lock.unlock()

Review comment:
       it can't be removed, as we aim to release ingest lock once complete tag 
'loading' for the choosed stage.

##########
File path: 
core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java
##########
@@ -1521,6 +1521,10 @@ private CarbonCommonConstants() {
 
   public static final String CARBON_QUERY_STAGE_INPUT_DEFAULT = "false";
 
+  public static final String CARBON_INSERT_STAGE_TIMEOUT = 
"carbon.insert.stage.timeout";
+
+  public static final long CARBON_INSERT_STAGE_TIMEOUT_DEFAULT = 28800000;

Review comment:
       modified




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Reply via email to