steveloughran commented on a change in pull request #1003: HADOOP-16384: Avoid
inconsistencies between DDB and S3
URL: https://github.com/apache/hadoop/pull/1003#discussion_r302149768
##########
File path: hadoop-tools/hadoop-aws/pom.xml
##########
@@ -186,7 +186,11 @@
<exclude>**/ITestS3AHuge*.java</exclude>
<!-- this sets out to overlaod DynamoDB, so must be run
standalone -->
<exclude>**/ITestDynamoDBMetadataStoreScale.java</exclude>
+ <!-- MR jobs spawn enough processes that they use up all
RAM -->
Review comment:
1. The terasort tests are *also* MR jobs; I consider the comment to be
correctly placed. Sorry.
2. The issue isn't that the minicluster uses lots of ram, it is that we span
multiple processes as part of the job (this is good) and together they use up
all your ram. If on a parallel test run >1 MR test suite is run at the same
time, your machine pages out the VM, processes pause, tests timeout, my video
on conf calls breaks up etc. Doing it sequentially ensures that the machine
stays usable during a test run of, say, threads=8
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
