dengzhhu653 commented on code in PR #3836:
URL: https://github.com/apache/hive/pull/3836#discussion_r1084142900
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/TestFileSinkOperator.java:
##########
@@ -183,37 +183,42 @@ public void testNonAcidRemoveDuplicate() throws Exception
{
JobConf jobConf2 = new JobConf(jobConf);
jobConf2.set("mapred.task.id", "000000_1");
- FileSinkOperator speculative = (FileSinkOperator)OperatorFactory.get(
+ FileSinkOperator op2 = (FileSinkOperator)OperatorFactory.get(
new CompilationOpContext(), FileSinkDesc.class);
- speculative.setConf(desc);
- speculative.initialize(jobConf2, new ObjectInspector[]{inspector});
+ op2.setConf(desc);
+ op2.initialize(jobConf2, new ObjectInspector[]{inspector});
for (Object r : rows) {
op1.process(r, 0);
- speculative.process(r, 0);
+ op2.process(r, 0);
}
op1.close(false);
- // speculative task also ends successfully
- speculative.close(false);
+ // Assume op2 also ends successfully, this happens in different containers
+ op2.close(false);
Path[] paths = findFilesInBasePath();
List<Path> mondays = Arrays.stream(paths)
.filter(path ->
path.getParent().toString().endsWith("partval=Monday/HIVE_UNION_SUBDIR_0"))
.collect(Collectors.toList());
- Assert.assertTrue(mondays.size() == 2);
+ Assert.assertEquals("Two result files were created", 2, mondays.size());
Set<String> fileNames = new HashSet<>();
fileNames.add(mondays.get(0).getName());
fileNames.add(mondays.get(1).getName());
- Assert.assertTrue(fileNames.contains("000000_1") &&
fileNames.contains("000000_0"));
+ Assert.assertTrue("000000_1 file is expected",
fileNames.contains("000000_1"));
+ Assert.assertTrue("000000_0 file is expected",
fileNames.contains("000000_0"));
+
+ // This happens in HiveServer2 when the job is finished, the job will call
Review Comment:
yeah, the `closeOp(boolean abort)` is called from task, but the `jobCloseOp`
is called from the jobclient side(jobClose-> jobCloseOp):
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java#L732-L748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For additional commands, e-mail: gitbox-h...@hive.apache.org
