[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288590#comment-17288590
]
Eric Badger commented on MAPREDUCE-7320:
I think leaving the test logs around is a feature, not a bug. I agree with
[~Jim_Brennan] on keeping them around and deleting them at the start of the
next run of unit tests
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288425#comment-17288425
]
Ahmed Hussein commented on MAPREDUCE-7320:
--
You have a reasonable point [~Jim_Brennan]. I agree it would be inconvenient to
modify the code just to look to the output and the logs.
I will disable the cleaning in {{@AfterClass}}.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288420#comment-17288420
]
Jim Brennan commented on MAPREDUCE-7320:
I would prefer to cleanup at the start of the test. I know we have had cases
in the past where I needed to look at these logs after our automated unit test
builds. I wouldn't want to have to modify the code to enable that.
I am curious how others feel about this?
cc: [~epayne], [~jeagles], [~ebadger], [~jhung]
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287438#comment-17287438
]
Ahmed Hussein commented on MAPREDUCE-7320:
--
Thanks [~Jim_Brennan].
I see your point.
My thoughts were that leaving the generated data by default increase the disk
utilization.
I face this problem because local hadoop repos sitting on my local machine are
eating large space, especially when I run long sequence of unit tests.
My intuition was that the common case is to run the unit tests, while debugging
output file is less frequent generally speaking. In that case case, the
developer can omit the {{@AfterClass}} that deletes the directory until he is
done with debugging.
P.S: I also found many unit tests that generate data in the home user directory
{{/tmp/hadoop}} which usually never get cleaned. This probably can be a
separate jira to organize the output.
Let me know if you still prefer to keep the data output after the unit test is
done. As you said, having the top level predictable is a good goal by itself.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17287385#comment-17287385
]
Jim Brennan commented on MAPREDUCE-7320:
[~ahussein] Thanks for cleaning this up. But I always thought keeping these
around was intentional? If you get a failure, it is sometimes useful to be
able to find the logs of the mini cluster (especially job/container logs).
In cases like this, you would want to make sure the top level directory is
predictable (which you have done), but you would want to delete them at the
beginning of the test instead of the end.
This would still leave a bunch of files around after running the unit tests,
but a subsequent run would first delete the old stuff. What do you think?
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286079#comment-17286079
]
Ahmed Hussein commented on MAPREDUCE-7320:
--
This fix is cleaning up the clusters directories after finishing the unit tests.
* It touches {{MiniYARNCluster.java}} in order to change the base dir to
{{target/test-dir/$TEST_CLASS_NAME}}
* test classes affected
** TestMRJobClient,
** TestStreamingBadRecords,
** TestClusterMapReduceTestCase,
** TestBadRecords.
** TestMRCJCJobClient,
** TestJobName
I tested the TestUnits that use {{MiniYARNCluster.java}} such as:
{code:bash}
Class
MiniYARNCluster
TestOSSMiniYarnCluster (3 usages found)
TestMRTimelineEventHandling (4 usages found)
TestJobHistoryEventHandler (3 usages found)
TestHadoopArchiveLogs (3 usages found)
TestHadoopArchiveLogsRunner (3 usages found)
TestDynamometerInfra (3 usages found)
TestDSTimelineV10
TestDSTimelineV20
TestDSTimelineV15
TestUnmanagedAMLauncher (3 usages found)
TestApplicationMasterServiceProtocolForTimelineV2
TestFederationRMFailoverProxyProvider (3 usages found)
TestHedgingRequestRMFailoverProxyProvider (4 usages found)
TestNoHaRMFailoverProxyProvider (5 usages found)
TestRMFailover (4 usages found)
TestAMRMClient
TestAMRMClientPlacementConstraints
TestAMRMProxy (5 usages found)
TestNMClient (3 usages found)
TestOpportunisticContainerAllocationE2E (3 usages found)
TestYarnClient (3 usages found)
TestYarnClientWithReservation (12 usages found)
TestYarnCLI (7 usages found)
TestContainerManagerSecurity (2 usages found)
TestDiskFailures (2 usages found)
TestMiniYarnCluster (9 usages found)
TestMiniYARNClusterForHA (2 usages found)
TestMiniYarnClusterNodeUtilization (3 usages found)
{code}
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
