Sanjay and I had a discussion on this one earlier. We thought that this would help Namenode robustness. We also thought that this was part of Java 6, and we could make this feature optionally configurable.
Other resource limitations (other than memory) are CPU, network and disk. We thought that we do not need to monitor those resources. The monitoring of critical resources and the policy of what action to take can be outside the actual Namenode process itself. Thanks, dhruba -----Original Message----- From: Doug Cutting [mailto:[EMAIL PROTECTED] Sent: Monday, December 03, 2007 2:04 PM To: hadoop-dev@lucene.apache.org Subject: limiting memory use on namenode? Over on pig-dev, Sam Pullara pointed out the following Java mechanism that permits one to be notified when memory usage exceeds a threshold. http://java.sun.com/javase/6/docs/api/java/lang/management/MemoryPoolMXB ean.html#UsageThreshold Perhaps we could use something like this on the namenode? For example, when memory usage is too high, the namenode could refuse creation of new files. This would still crash applications, but it would keep the filesystem itself from crashing in a way that is hard to recover while folks remove excessive (presumably) small files. Doug
