[
https://issues.apache.org/jira/browse/MAPREDUCE-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783643#action_12783643
]
Iyappan Srinivasan commented on MAPREDUCE-1140:
-----------------------------------------------
| Regarding the tests, I spoke offline to Amarsri to understand the scenario
executed by Karthikeyan in the comment above. It was not very clear why file1
was being added twice. Some more details on configuration - that it was run on
a single node, max failures was set to 1 should be documented for better
understanding.
- Configuration :
The tests run on a single node running JT/NN and another node running TT/DN.
map.max.attempts is set to 1 and reduce.max.attempts is also set to 1 and
local.cache.size is set to 4 GB and mapred.local.dir is set to only 1 spindle
and not all the spindles.
This is done to force the TT to localize in the same path and try deleting the
localcaheFiles when the size exceeds 4 GB.
The idea of the above testcase is :
Ran Job1 with cache files file1 and file2 - Job succeeded.
Ran Job2 with cache files file3 and file1. When file3 is getting localized,
removed file3 from dfs - Job2 failed.
- Here since file3 is deleted, the reference count of file1 should not be
decremented twice(once during setup and once during cleanup).Thats the
objective of this scenario.
Ran Job3 with cache files file1, file1(again) and file4. file4 is huge (say
5GB), larger than local.cache.size.
- To make sure that the decrement happened properly, file1 is added twice.
When file4 is added, which
is more than local cache size, other files like file2 and file3 ( which were
used in the previous jobs) gets deleted
but not file1 (because it had reference count proper ).
| In order to match the regressions tests in trunk, I would suggest we also
have in Job 2 a file, say file5, which we should verify is not even localized
(because file3 fails localization). Then we can include file 5 in Job 3 and
make sure localization happens successfully.
This scenario is tested and the localization happens successfully.
> Per cache-file refcount can become negative when tasks release
> distributed-cache files
> --------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1140
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1140
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.20.2, 0.21.0, 0.22.0
> Reporter: Vinod K V
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.22.0
>
> Attachments: patch-1140-1.txt, patch-1140-2-ydist.txt,
> patch-1140-2.txt, patch-1140-3.txt, patch-1140-ydist.txt,
> patch-1140-ydist.txt, patch-1140.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
