[
https://issues.apache.org/jira/browse/HADOOP-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14549616#comment-14549616
]
Allen Wittenauer commented on HADOOP-11984:
-------------------------------------------
bq. Is it worthwhile to trade the additional complexity and possibly reduced
robustness for the optimized execution time?
I don't find the current code complex at all compared to some of the gymnastics
happening in hadoop-functions.sh. I might have a different opinion if the
docker bits ever get committed.
bq. The UNIX world supports bash pretty well. However, Hadoop officially
supports Windows as well. How does the compatibility story work for Windows
folks? Unfortunately things like path separators are quite annoying to handle.
We've already done it for the client code in trunk. It's likely better than
the DOS batch code. My hunch is that it won't be that bad to get test-patch.sh
running under cygwin. More complex than the client code though.
bq. What I'm trying to explore here from a maintainability / productivity
prospective whether we should allow writing supplement scripts in languages
that have a wide range of libraries to take care of things like text processing
/ OS manipulation, regardless it's Python, Perl or other languages.
Until the portability issue gets dealt with (which will happen as more time
passes), the answer here is likely no without potentially significantly
increasing the burden of the installed software set.
Of course, unless someone is actually going to do the work, this whole
conversation is sort of moot. Traditionally, the non-java parts of Hadoop are
regularly ignored until it impacts someone's corporate goals. I don't expect
that to change any time soon.
bq.I can't repro the OutOfMemoryError problems after setting reuseForks to
false. I don't know if I'm just overloading these hosts by running multiple
processes with a maximum 1 GB heap. I don't have much visibility into these
hosts. If anyone else out there knows how to find host-level details, please
let me know. Otherwise, I'll probably need to coordinate with infra.
Hmm. I wonder if the fact that MAVEN_OPTS is set to "-Xmx3072m
-XX:MaxPermSize=768m" prior to test-patch.sh invocation in the HADOOP precommit
is part of the problem here.
This is where it'd be super useful to run precommit in the Docker container so
we could see if it was actually a maven version problem. :(
> Enable parallel JUnit tests in pre-commit.
> ------------------------------------------
>
> Key: HADOOP-11984
> URL: https://issues.apache.org/jira/browse/HADOOP-11984
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Attachments: HADOOP-11984.001.patch, HADOOP-11984.002.patch,
> HADOOP-11984.003.patch, HADOOP-11984.004.patch, HADOOP-11984.005.patch
>
>
> HADOOP-9287 and related issues implemented the parallel-tests Maven profile
> for running JUnit tests in multiple concurrent processes. This issue
> proposes to activate that profile during pre-commit to speed up execution.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
