stack-3 wrote: > > > Yes. Could have been deadlocked. Next time send the "-QUIT" signal and > you'll get a threaddump in the regionserver .out file. > >
Ok. I'll try this if the case happens again. stack-3 wrote: > > > Disks full on your HDFS? > > Yes. One of the disk is full. Disk size are not heterogeneous. Moreover I tried to keep a part of space free for other usage, using the following hadoop directives: <property> <name>dfs.datanode.du.reserved</name> <value>4000000000</value> <description>Reserved space in bytes per volume. Always leave this much space free for non dfs use. </description> </property> But free space goes under 1 Gb, as if I kept the default settings Then I added <property> <name>dfs.datanode.du.pct</name> <value>0.9f</value> <description>When calculating remaining space, only use this percentage of the real available space </description> </property> I removed some stuff, but no changes. It falls under 1Gb (I expected to have 3Gb free at least) Now df -h gives me Filesystem Size Used Avail Use% Mounted on /dev/sda1 36G 34G 602M 99% / I saw in hadoop jira that the implementation of these directives was subject to discussions. Or is that possible that hbase client overrides the hadoop configuration ? Anyway, I'll post a messge in their board. stack-3 wrote: > > > I see above too on occasion on our clusters. Was going to spend some > time on it today. In particular, see if HADOOP-3831 helps. > > 1GB of RAM as Jon Gray has suggested is not going to cut it in my > experience. With so little memory, would suggest you run datanodes and > regionservers on their own machines. > > While I'm giving out advice, smile, be sure to have upped your ulimit so > hbase can use > 1024 file descriptors and run with DEBUG enabled so you > can see more about why hbase is doing (See FAQ for how to do both). > > Thanks J-A, > St.Ack > > File descriptors limit is set as FAQ said. I'll fix the HD free space problem, and distribute process and memory differently on my cluster, following you and Jonathan's advise It should help. Thanks a lot. Have a nice day -- View this message in context: http://www.nabble.com/Dataloss-HBASE-728-tp20013732p20030496.html Sent from the HBase User mailing list archive at Nabble.com.
