[jira] [Commented] (HADOOP-10613) Potential Resource Leaks in FileSystem.CACHE
[ https://issues.apache.org/jira/browse/HADOOP-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001518#comment-14001518 ] Nemon Lou commented on HADOOP-10613: Yes,application is responsable to release FileSystem object int the case i described above for now. But i think FileSystem can do better(easier to use). There are differences between hadoop's FileSystem and java's File. For java File object,user need to close it very time after open it. For hadoop FileSystem object ,user don't need to close it after call FileSystem.get(conf).Because the FileSystem object is cached and can be reused next time when user's code invoke FileSystem.get(conf) again.It's life cycle is higher than java's File object. The perpose to call FileSystem's close method is removing it from cache ,and making delete-on-exit featrue work. If user' code never call FileSystem's close method,it's still ok.Because there is a shut down hook for each FileSystem object,the close method will be called when application stops.The only problem is the memory leak.So,a LRU cache or something similar will be better than the no size limited hashmap based cache. Potential Resource Leaks in FileSystem.CACHE - Key: HADOOP-10613 URL: https://issues.apache.org/jira/browse/HADOOP-10613 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.4.0 Reporter: Nemon Lou There is no size limit of the hashmap in FileSystem.CACHE, which can cause a potential memory leak. If every time i use a new UGI object to invoke FileSystem.get(conf) and never invoke FileSystem's close method,this issue will raise. If there is a size limit of the hashmap or changing fileSystem instances to soft reference,then user's code don't need to consider too much about the cache leak issues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HADOOP-10613) Potential Resource Leaks in FileSystem.CACHE
Nemon Lou created HADOOP-10613: -- Summary: Potential Resource Leaks in FileSystem.CACHE Key: HADOOP-10613 URL: https://issues.apache.org/jira/browse/HADOOP-10613 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 2.4.0 Reporter: Nemon Lou There is no size limit of the hashmap in FileSystem.CACHE, which can cause a potential memory leak. If every time i use a new UGI object to invoke FileSystem.get(conf) and never invoke FileSystem's close method,this issue will raise. If there is a size limit of the hashmap or changing fileSystem instances to soft reference,then user's code don't need to consider too much about the cache leak issues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HADOOP-9655) Connection object in IPC Client can not run concurrently during connection time out
[ https://issues.apache.org/jira/browse/HADOOP-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nemon Lou updated HADOOP-9655: -- Summary: Connection object in IPC Client can not run concurrently during connection time out (was: IPC Client call to the same host with multi thread takes very long time to report connection time out for many times ) Connection object in IPC Client can not run concurrently during connection time out --- Key: HADOOP-9655 URL: https://issues.apache.org/jira/browse/HADOOP-9655 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.0.4-alpha Reporter: Nemon Lou Attachments: HADOOP-9655.patch When one machine power off during running a job ,MRAppMaster find tasks timed out on that host and then call stop container for each container concurrently. But the IPC layer did it serially, for each call,the connection time out exception toke a few minutes to raise after 45 times reties. And AM hang for many hours to wait for stopContainer to finish. The jstack output file shows that most threads stuck at Connection.addCall waiting for a lock object hold by Connection.setupIOstreams. (The setupIOstreams method run slowlly becauseof connection time out during setupconnection.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9655) IPC Client call to the same host with multi thread takes very long time to report connection time out for many times
[ https://issues.apache.org/jira/browse/HADOOP-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705506#comment-13705506 ] nemon lou commented on HADOOP-9655: --- This patch has been tested on my cluster and has solved the problem. IPC Client call to the same host with multi thread takes very long time to report connection time out for many times - Key: HADOOP-9655 URL: https://issues.apache.org/jira/browse/HADOOP-9655 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9655.patch When one machine power off during running a job ,MRAppMaster find tasks timed out on that host and then call stop container for each container concurrently. But the IPC layer did it serially, for each call,the connection time out exception toke a few minutes to raise after 45 times reties. And AM hang for many hours to wait for stopContainer to finish. The jstack output file shows that most threads stuck at Connection.addCall waiting for a lock object hold by Connection.setupIOstreams. (The setupIOstreams method run slowlly becauseof connection time out during setupconnection.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9655) IPC Client call to the same host with multi thread takes very long time to report connection time out for many times
[ https://issues.apache.org/jira/browse/HADOOP-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9655: -- Attachment: HADOOP-9655.patch This patch use a different object for wait and notify ,so one thread invoking addCall method won't be blocked by another thread calling setupConnection method. IPC Client call to the same host with multi thread takes very long time to report connection time out for many times - Key: HADOOP-9655 URL: https://issues.apache.org/jira/browse/HADOOP-9655 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9655.patch When one machine power off during running a job ,MRAppMaster find tasks timed out on that host and then call stop container for each container concurrently. But the IPC layer did it serially, for each call,the connection time out exception toke a few minutes to raise after 45 times reties. And AM hang for many hours to wait for stopContainer to finish. The jstack output file shows that most threads stuck at Connection.addCall waiting for a lock object hold by Connection.setupIOstreams. (The setupIOstreams method run slowlly becauseof connection time out during setupconnection.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9655) IPC Client call to the same host with multi thread takes very long time to report connection time out for many times
[ https://issues.apache.org/jira/browse/HADOOP-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9655: -- Status: Patch Available (was: Open) IPC Client call to the same host with multi thread takes very long time to report connection time out for many times - Key: HADOOP-9655 URL: https://issues.apache.org/jira/browse/HADOOP-9655 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9655.patch When one machine power off during running a job ,MRAppMaster find tasks timed out on that host and then call stop container for each container concurrently. But the IPC layer did it serially, for each call,the connection time out exception toke a few minutes to raise after 45 times reties. And AM hang for many hours to wait for stopContainer to finish. The jstack output file shows that most threads stuck at Connection.addCall waiting for a lock object hold by Connection.setupIOstreams. (The setupIOstreams method run slowlly becauseof connection time out during setupconnection.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9655) IPC Client call to the same host with multi thread takes very long time to report connection time out for many times
nemon lou created HADOOP-9655: - Summary: IPC Client call to the same host with multi thread takes very long time to report connection time out for many times Key: HADOOP-9655 URL: https://issues.apache.org/jira/browse/HADOOP-9655 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 2.0.4-alpha Reporter: nemon lou When one machine power off during running a job ,MRAppMaster find tasks timed out on that host and then call stop container for each container concurrently. But the IPC layer did it serially, for each call,the connection time out exception toke a few minutes to raise after 45 times reties. And AM hang for many hours to wait for stopContainer to finish. The jstack output file shows that most threads stuck at Connection.addCall waiting for a lock object hold by Connection.setupIOstreams. (The setupIOstreams method run slowlly becauseof connection time out during setupconnection.) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Attachment: HADOOP-9512.patch Changing JobDiagnoser into a normal hadoop tool (extends Configured implements Tool). Also fixing bugs for supporting hdfs. Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch, HADOOP-9512.patch, HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Status: Patch Available (was: Open) Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch, HADOOP-9512.patch, HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Status: Open (was: Patch Available) Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch, HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Attachment: HADOOP-9512.patch Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Status: Patch Available (was: Open) Adding hadoop vaidya maven project with new history parser. Most of the code is the same with hadoop vaidya 1.0. We can use bin/yarn jar share/hadoop/tools/lib/hadoop-vaidya-3.0.0-SNAPSHOT.jar to run this tool. Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Status: Open (was: Patch Available) Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Attachment: HADOOP-9512.patch Fix the warnings Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch, HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9512) Add Hadoop-Vaidya to branch2
[ https://issues.apache.org/jira/browse/HADOOP-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-9512: -- Status: Patch Available (was: Open) Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Attachments: HADOOP-9512.patch, HADOOP-9512.patch Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9512) Add Hadoop-Vaidya to branch2
nemon lou created HADOOP-9512: - Summary: Add Hadoop-Vaidya to branch2 Key: HADOOP-9512 URL: https://issues.apache.org/jira/browse/HADOOP-9512 Project: Hadoop Common Issue Type: New Feature Components: tools Affects Versions: 2.0.4-alpha Reporter: nemon lou Hadoop-Vaidya exists in hadoop1.0 series and we need to add it to hadoop branch2. The only changes for supporting Hadoop2 are job history perser and job counters ,etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9498) ActiveStandbyElector doesn't work with secure ZooKeeper
nemon lou created HADOOP-9498: - Summary: ActiveStandbyElector doesn't work with secure ZooKeeper Key: HADOOP-9498 URL: https://issues.apache.org/jira/browse/HADOOP-9498 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.3-alpha Reporter: nemon lou When security is enabled for ZooKeeper, client will receive SaslAuthenticated or AuthFailed event from server side. As for now ActiveStandbyElector does not know these events ,it then raise fatalError and exit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8602) Passive mode support for FTPFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-8602: -- Status: Patch Available (was: Open) Passive mode support for FTPFileSystem -- Key: HADOOP-8602 URL: https://issues.apache.org/jira/browse/HADOOP-8602 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: nemon lou Priority: Minor Attachments: HADOOP-8602.patch, HADOOP-8602.patch FTPFileSystem uses active mode for default data connection mode.We shall be able to choose passive mode when active mode doesn't work (firewall for example). My thoughts is to add an option fs.ftp.data.connection.mode in core-site.xml.Since FTPClient(in org.apache.commons.net.ftp package) already supports passive mode, we just need to add a few code in FTPFileSystem .connect() method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8602) Passive mode support for FTPFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nemon lou updated HADOOP-8602: -- Attachment: HADOOP-8602.patch Passive mode support for FTPFileSystem -- Key: HADOOP-8602 URL: https://issues.apache.org/jira/browse/HADOOP-8602 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 1.0.3, 2.0.0-alpha Reporter: nemon lou Priority: Minor Attachments: HADOOP-8602.patch, HADOOP-8602.patch FTPFileSystem uses active mode for default data connection mode.We shall be able to choose passive mode when active mode doesn't work (firewall for example). My thoughts is to add an option fs.ftp.data.connection.mode in core-site.xml.Since FTPClient(in org.apache.commons.net.ftp package) already supports passive mode, we just need to add a few code in FTPFileSystem .connect() method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8602) Passive mode support for FTPFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13421993#comment-13421993 ] nemon lou commented on HADOOP-8602: --- I havn't found any test case for FTPFileSystem.java. So no test case added for passive mode option. I will add one if necessary. Passive mode support for FTPFileSystem -- Key: HADOOP-8602 URL: https://issues.apache.org/jira/browse/HADOOP-8602 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 1.0.3, 2.0.0-alpha Reporter: nemon lou Priority: Minor Attachments: HADOOP-8602.patch FTPFileSystem uses active mode for default data connection mode.We shall be able to choose passive mode when active mode doesn't work (firewall for example). My thoughts is to add an option fs.ftp.data.connection.mode in core-site.xml.Since FTPClient(in org.apache.commons.net.ftp package) already supports passive mode, we just need to add a few code in FTPFileSystem .connect() method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8602) Passive mode support for FTPFileSystem
nemon lou created HADOOP-8602: - Summary: Passive mode support for FTPFileSystem Key: HADOOP-8602 URL: https://issues.apache.org/jira/browse/HADOOP-8602 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 2.0.0-alpha, 1.0.3 Reporter: nemon lou Priority: Minor FTPFileSystem uses active mode for default data connection mode.We shall be able to choose passive mode when active mode doesn't work (firewall for example). My thoughts is to add an option fs.ftp.data.connection.mode in core-site.xml.Since FTPClient(in org.apache.commons.net.ftp package) already supports passive mode, we just need to add a few code in FTPFileSystem .connect() method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
