[
https://issues.apache.org/jira/browse/HIVE-14540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423191#comment-15423191
]
Siddharth Seth commented on HIVE-14540:
---------------------------------------
bq. AFAIK maven is currently configured to fork a jvm for every
test(resuseforks=false)..
That was a little unexpected, or not... this needs to be fixed as well.
https://docs.google.com/spreadsheets/d/1QYLWp61kLWXEm6MGxwFrG5caTiTREm8SJu4NsXSspF8/edit?usp=sharing
In a ptest run reuseForks=false does not make a lot of difference (yet). Each
individual test class is run via an independent mvn test invocation. That's
over 600 mvn test invocations on 24 drones. The intent of batching was to
amortize some of the time wasted in doing this. If we go down the batching
route, reuseForks will come into play.
This sheet contains information on test runtimes - reported by junit vs
reported by ptest.
https://docs.google.com/spreadsheets/d/1QYLWp61kLWXEm6MGxwFrG5caTiTREm8SJu4NsXSspF8/edit?usp=sharing
bq. So...I would like to suggest to explore a different path and try to disable
this feature in most of the modules, and identify and move the "bad" testcases
to some integration module, where they can run isolated - after this someone
may dig deeper into those bad cases and possibly fix them (and move them back
Did not really consider this, since I just assumed that the tests were in a
state where running them on a single node (as a single process) would take too
long. As part of HIVE-13503, there's an effort to identify individual tests
which run for a long time, and fix them. We should definitely explore running
tests per-module and identify modules which can all be run as a single batch.
Also 'unit' tests which run for far too long.
> Create batches for non qfile tests
> ----------------------------------
>
> Key: HIVE-14540
> URL: https://issues.apache.org/jira/browse/HIVE-14540
> Project: Hive
> Issue Type: Sub-task
> Reporter: Siddharth Seth
> Assignee: Siddharth Seth
>
> From run 790:
> Reported runtime by junit: 17 hours
> Reported runtime by ptest: 34 hours
> A lot of time is wasted spinning up mvn test for each individual test, which
> otherwise takes less than 1 second. These tests could end up taking 20-30
> seconds. Combined with HIVE-14539 - 60-70s.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
