Hi Brian, Thanks for your response.. Actually the master process doesn't even try to access HDFS.. What's happening is that process Master forks 2 processes A and B. A and B are the ones that try to connect to HDFS.. I tried doing this in a separate application and it worked.. for some reason it doesn't work in the other application that I'm working on. Since I'm not connecting from master then I cannot really apply any of your advices.. do you have other ideas? Thanks
On Fri, Mar 23, 2012 at 7:25 AM, Brian Bockelman <bbock...@cse.unl.edu>wrote: > Hi Tareq, > > This is because libhdfs will keep a bit of state data (especially if the > master was connected to HDFS). > > Three suggestions: > 1) [Likely to work] Fork the children first, then do any HDFS actions in > the master. Alternately, don't have the master do any HDFS actions; have > it fork a child which does them for it. > 2) [May work] Try disconnecting the master from HDFS, then forking the > children. > 3) [If 2 doesn't work] Have the master disconnect, then fork the children, > then have the children exec. > > Brian > > On Mar 22, 2012, at 11:38 PM, Tareq Aljabban wrote: > > > Hi, > > I'm using libhdfs C interface to access HDFS. > > The application accessing HDFS has a master process that forks many > > processes, and these processes try to connect to HDFS. > > Upon connection, one of the processes is exiting as soon as it reaches > the > > hdfsConnect(). > > I tried simulating this behavior by creating a separate application where > > the master process forks two processes that connect to HDFS and send > > requests.. this worked without any problem.. so I got confused.. what's > > really causing the connection problems then? > > Any insights on this is much appreciated. > >
