ayushtkn commented on a change in pull request #2861:
URL: https://github.com/apache/hadoop/pull/2861#discussion_r611748559
##########
File path:
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java
##########
@@ -628,8 +629,15 @@ public void testDistCpWithIterator() throws Exception {
GenericTestUtils
.createFiles(remoteFS, source, getDepth(), getWidth(), getWidth());
+ GenericTestUtils.LogCapturer log =
Review comment:
Thanx @steveloughran, do you suggest that we should have two options
like -useiteratorforsource and -useiteratorfortarget. Do you think in that case
we would be able to save out on memory? since the target list is being build as
part of CopyCommitter, so even if one takes the normal path, We would get OOM,
just `when` will differ?
Regarding the log stuff, That was the only thing I could think of, to
confirm if iterator was used. And during migration to Log4J2, Will moving to
something like this will also be of no help:
https://github.com/apache/hive/blob/master/ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java#L285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
