I am accessing through threads in parallel. What is the concept of Lease in HDFS??
Regards, JD ________________________________ From: Harsh J <[email protected]> To: jagaran das <[email protected]> Sent: Friday, 5 August 2011 11:37 PM Subject: Re: java.io.IOException: config() How long are you keeping it open for? On 06-Aug-2011, at 10:14 AM, jagaran das wrote: Hi, > > >I am using CDH3. >I need to stream huge amount of data from our application to hadoop. >I am opening a connection like > >config.set("fs.default.name",hdfsURI); >FileSystem dfs = FileSystem.get(config); >String path = hdfsURI + connectionKey; >Path destPath = new Path(path); >logger.debug("Path -- " + destPath.getName()); >outStream = dfs.create(destPath); >and keeping the outStream open for some time and writing continuously through >it and then closing it. >But it is throwing > > > >5Aug2011 21:36:48,550 DEBUG >[LeaseChecker@DFSClient[clientName=DFSClient_218151655, ugi=jagarandas]: >java.lang.Throwable: for testing >at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.toString(DFSClient.java:1181) >at org.apache.hadoop.util.Daemon.<init>(Daemon.java:38) >at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.put(DFSClient.java:1094) >at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:547) >at >org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:219) >at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:584) >at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:565) >at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:472) >at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:464) >at >com.apple.ireporter.common.persistence.ConnectionManager.createConnection(ConnectionManager.java:66) >at >com.apple.ireporter.common.persistence.HDPPersistor.writeToHDP(HDPPersistor.java:93) >at >com.apple.ireporter.datatransformer.translator.HDFSTranslator.persistData(HDFSTranslator.java:41) >at >com.apple.ireporter.datatransformer.adapter.TranslatorAdapter.processData(TranslatorAdapter.java:61) >at >com.apple.ireporter.datatransformer.DefaultMessageListener.persistValidatedData(DefaultMessageListener.java:276) >at >com.apple.ireporter.datatransformer.DefaultMessageListener.onMessage(DefaultMessageListener.java:93) >at >org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:506) >at >org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:463) >at >org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:435) >at >org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:322) >at >org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:260) >at >org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:944) >at >org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:868) >at java.lang.Thread.run(Thread.java:680) >] (RPC.java:230) - Call: renewLease 4 >05Aug2011 21:36:48,550 DEBUG [listenerContainer-1] (DFSClient.java:3274) - >DFSClient writeChunk allocating new packet seqno=0, >src=/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819, > packetSize=65557, chunksPerPacket=127, bytesCurBlock=0 >05Aug2011 21:36:48,551 DEBUG [Thread-11] (DFSClient.java:2499) - Allocating >new block >05Aug2011 21:36:48,552 DEBUG [sendParams-0] (Client.java:761) - IPC Client >(47) connection to localhost/127.0.0.1:8020 from jagarandas sending #3 >05Aug2011 21:36:48,553 DEBUG [IPC Client (47) connection to >localhost/127.0.0.1:8020 from jagarandas] (Client.java:815) - IPC Client (47) >connection to localhost/127.0.0.1:8020 from jagarandas got value #3 >05Aug2011 21:36:48,556 DEBUG [Thread-11] (RPC.java:230) - Call: addBlock 4 >05Aug2011 21:36:48,557 DEBUG [Thread-11] (DFSClient.java:3094) - pipeline = >127.0.0.1:50010 >05Aug2011 21:36:48,557 DEBUG [Thread-11] (DFSClient.java:3102) - Connecting to >127.0.0.1:50010 >05Aug2011 21:36:48,559 DEBUG [Thread-11] (DFSClient.java:3109) - Send buf size >131072 >05Aug2011 21:36:48,635 DEBUG [DataStreamer for file >/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819 > block blk_-5183404460805094255_1042] (DFSClient.java:2533) - DataStreamer >block blk_-5183404460805094255_1042 wrote packet seqno:0 size:1522 >offsetInBlock:0 lastPacketInBlock:true >05Aug2011 21:36:48,638 DEBUG [ResponseProcessor for block >blk_-5183404460805094255_1042] (DFSClient.java:2640) - DFSClient Replies for >seqno 0 are SUCCESS >05Aug2011 21:36:48,639 DEBUG [DataStreamer for file >/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_247811312605307819 > block blk_-5183404460805094255_1042] (DFSClient.java:2563) - Closing old >block blk_-5183404460805094255_1042 >05Aug2011 21:36:48,645 DEBUG [sendParams-0] (Client.java:761) - IPC Client >(47) connection to localhost/127.0.0.1:8020 from jagarandas sending #4 >05Aug2011 21:36:48,647 DEBUG [IPC Client (47) connection to >localhost/127.0.0.1:8020 from jagarandas] (Client.java:815) - IPC Client (47) >connection to localhost/127.0.0.1:8020 from jagarandas got value #4 >05Aug2011 21:36:48,648 DEBUG [listenerContainer-1] (RPC.java:230) - Call: >complete 3 > > >Please help as it a production enhancement for us. > > >Regards >Jagaran > > > >________________________________ >From: Harsh J <[email protected]> >To: [email protected]; jagaran das <[email protected]> >Sent: Friday, 5 August 2011 8:54 PM >Subject: Re: java.io.IOException: config() > >Could you explain how/where you're stuck? > >That DEBUG log doesn't even seem like a valid throw; its just to get a >strace I believe. > >On Sat, Aug 6, 2011 at 8:52 AM, jagaran das <[email protected]> wrote: >> Hi, >> >> I have been struck with this exception: >> >> java.io.IOException: config() >> at org.apache.hadoop.conf.Configuration.(Configuration.java:211) >> at org.apache.hadoop.conf.Configuration.(Configuration.java:198) >> at >> org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:99) >> at test.TestApp.main(TestApp.java:19) >> >> >> >> 05Aug2011 20:08:53,303 DEBUG >> [LeaseChecker@DFSClient[clientName=DFSClient_-1591195062, >> ugi=jagarandas,staff,com.apple.sharepoint.group.1,_developer,_lpoperator,_lpadmin,_appserveradm,admin,_appserverusr,localaccounts,everyone,fmsadmin,com.apple.access_screensharing,com.apple.sharepoint.group.2,com.apple.sharepoint.group.3]: >> java.lang.Throwable: for testing >> >> >> 05Aug2011 20:08:53,315 DEBUG [listenerContainer-1] (DFSClient.java:3012) - >> DFSClient writeChunk allocating new packet seqno=0, >> src=/home/hadoop/listenerContainer-1jagaran-dass-macbook-pro.local_222812011-08-05-20-08-52, >> packetSize=65557, chunksPerPacket=127, bytesCurBlock=0 >> >> I saw the source code : >> >> public Configuration(boolean loadDefaults) { >> this.loadDefaults = loadDefaults; >> if (LOG.isDebugEnabled()) { >> LOG.debug(StringUtils.stringifyException(new IOException("config()"))); >> } >> synchronized(Configuration.class) { >> REGISTRY.put(this, null); >> } >> } >> >> Log is in debug mode. >> >> Can anyone please help me on this?? >> >> Regards, >> JD > > > >-- >Harsh J > > >
