Jeremy Pinkham wrote:
We are using the distributed cache in one of our jobs and have noticed
that the local copies on all of the task nodes never seem to get cleaned
up. Is there a mechanism in the API to tell the framework that those
copies are no longer needed so they can be deleted. I've tried using
releaseCache and deleting the source file from hdfs... but it still
remains in the local directories on each node
(<HADOOP_ROOT>/temp/hadoop-hadoop/mapred/local/taskTracker/archive
These files are shared by all the jobs. And DistributedCache does a lazy
deletion. So, releaseCache doesn't delete the files, it just decrements
the reference count.
The cache will be cleanedup when there is no space left, which can be
specified by configuration, "local.cache.size". i.e If the total size of
archive directory is more than allowed size (local.cache.size, default
value is 1MB), files with zero-reference count will get cleaned up. You
can specify a lower value for "local.cache.size" if needed.
Thanks
Amareshwari
Am I doing something wrong? not seeing the right API method to do the
cleaning? or is it intentional that these files must be removed
manually?
Thanks
jeremy
The information transmitted in this email is intended only for the person(s) or
entity to which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or taking
of any action in reliance upon, this information by persons or entities other
than the intended recipient is prohibited. If you received this email in error,
please contact the sender and permanently delete the email from any computer.