I have tried the way TestDFSShell.java does,
here's my code:
------------------------------------------------------------
public class CustomInterface
{
Configuration conf ;
FsShell fs ;
public CustomInterface()
{
conf = new Configuration();
fs = new FsShell();
fs.setConf(conf);
}
public int createDir(String strDirName,String strPath)
{
// omit exception catch
int iRet = 0;
strPath += strDirName;
String[] strCmd = new String[2];
strCmd[0] = "-mkdir";
strCmd[1] = strPath;
return m_fs.run(strCmd);
}
}
------------------------------------------------------------
Then i just call the createdir Method
for(int i =0 ; i < 100000 ; i ++)
{
custom.createDir("someName");
}
this cause the java vm process hold many threads
and these threads eat memory .
till the JVM Heap are eat up , throws Exceptions.
only larger Heap size holds more threads , but not fix the problem.
thanks.
Dhruba Borthakur wrote:
>
> One example of programmatically using FsShell is in
> src/test/org/apache/hadoop/dfs/TestDFSShell.java
>
> Thanks,
> dhruba
>
> -----Original Message-----
> From: KrzyCube [mailto:[EMAIL PROTECTED]
> Sent: Monday, July 23, 2007 7:49 PM
> To: [email protected]
> Subject: Calling FsShell.doMain() hold so many threads
>
>
> Hi there:
>
> i got two questions:
>
> Q1:
> I am try to call the FsShell.doMain() with my own code , which is
> only
> a easy wrapper of the FsShell.
> But when i am trying to create many dirs , 10000 etc. Exception like "Not
> enough memory for more threads" throw , i have set the -Xmx512m.
> Then i trying to view the process info while the program running ,
> then
> i found there are more and more threads invoked during the process , and
> eat
> more and more memory ,all threads still there without exit.
> Then i came to the source code , and found that while the
> FsShell.Main()
> for terminal call there is one line "System.exit(return_value_of_doMain)"
> ,
> Is that mean the call of the ToolBase.run() which implemented in
> FsShell.java is always create a new thread and have to be force terminated
> by System.exit() to kill the process ?
> So , if that is , how can i write my own code to use hadoop with
> FsShell
> in multi-thread mode , or is there any other way to do this ?
>
> Q2:
> I svn code , and run it in eclipse [the only reason i refer to
> eclipse
> is to indicate my environment],
> under Unbuntu 7.04.
> all about casual , i want to see how much time the FsShell.doMain()
> take , I use "new Date()" and
> get the interval with "DateEnd.getTime() - DateBeg.getTime()"
> Then i found that: even mkdir take more then 1000 [getTime shows]
> if there's no arguments , it take 25 , but even if i just give it a wrong
> argument , such as "-sl", it take more than 1000 , is that means the
> argument check take most of the time cost?
>
> --
> View this message in context:
> http://www.nabble.com/Calling-FsShell.doMain%28%29-hold-so-many-threads-tf41
> 33557.html#a11756139
> Sent from the Hadoop Users mailing list archive at Nabble.com.
>
>
>
>
--
View this message in context:
http://www.nabble.com/Calling-FsShell.doMain%28%29-hold-so-many-threads-tf4133557.html#a11759406
Sent from the Hadoop Users mailing list archive at Nabble.com.
