ArkoSharma commented on a change in pull request #1936:
URL: https://github.com/apache/hive/pull/1936#discussion_r580050588
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpWork.java
##########
@@ -184,42 +190,86 @@ public void setResultValues(List<String> resultValues) {
this.resultValues = resultValues;
}
- public List<Task<?>> externalTableCopyTasks(TaskTracker tracker, HiveConf
conf) {
+ public List<Task<?>> externalTableCopyTasks(TaskTracker tracker, HiveConf
conf) throws IOException {
if (conf.getBoolVar(HiveConf.ConfVars.REPL_DUMP_SKIP_IMMUTABLE_DATA_COPY))
{
return Collections.emptyList();
}
List<Task<?>> tasks = new ArrayList<>();
- while (externalTblCopyPathIterator.hasNext() && tracker.canAddMoreTasks())
{
- DirCopyWork dirCopyWork = new DirCopyWork(metricCollector,
currentDumpPath.toString());
- dirCopyWork.loadFromString(externalTblCopyPathIterator.next());
- Task<DirCopyWork> task = TaskFactory.get(dirCopyWork, conf);
- tasks.add(task);
- tracker.addTask(task);
- LOG.debug("added task for {}", dirCopyWork);
+ Retryable retryable = Retryable.builder()
+ .withHiveConf(conf)
+ .withRetryOnException(UncheckedIOException.class).build();
+ try {
+ retryable.executeCallable((Callable<Void>) ()-> {
+ try{
+ int numEntriesToSkip = tasks == null ? 0 : tasks.size();
+ while (externalTblCopyPathIterator.hasNext() &&
tracker.canAddMoreTasks()) {
+ if(numEntriesToSkip > 0) {
+ //skip tasks added in previous attempts of this retryable block
+ externalTblCopyPathIterator.next();
+ numEntriesToSkip--;
+ continue;
+ }
+ DirCopyWork dirCopyWork = new DirCopyWork(metricCollector,
currentDumpPath.toString());
+ dirCopyWork.loadFromString(externalTblCopyPathIterator.next());
+ Task<DirCopyWork> task = TaskFactory.get(dirCopyWork, conf);
+ tasks.add(task);
+ tracker.addTask(task);
Review comment:
They remain in memory in the same list : 'tasks'.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
