Which Hadoop version are you using exactly? I see you are setting dfs.datanode.data.dir which is a post 1.0 setting (from what I can tell by googling, since I didn't recognize it), but you are using a "hadoop-examples-1.0.3.jar" file that seems to imply you are on 1.0.3 which would probably not pick up dfs.datanode.data.dir
J-D On Tue, Sep 18, 2012 at 9:21 AM, Jason Huang <[email protected]> wrote: > I've done some more research but still can't start the HMaster node > (with similar error). Here is what I found in the Master Server log: > > Tue Sep 18 11:50:22 EDT 2012 Starting master on Jasons-MacBook-Pro.local > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > file size (blocks, -f) unlimited > max locked memory (kbytes, -l) unlimited > max memory size (kbytes, -m) unlimited > open files (-n) 65536 > pipe size (512 bytes, -p) 1 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 1064 > virtual memory (kbytes, -v) unlimited > > > 2012-09-18 11:50:23,306 INFO org.apache.hadoop.hbase.util.VersionInfo: > HBase 0.94.0 > 2012-09-18 11:50:23,306 INFO org.apache.hadoop.hbase.util.VersionInfo: > Subversion https://svn.apache.org/repos/asf/hbase/branches/0.94 -r > 1332822 > 2012-09-18 11:50:23,306 INFO org.apache.hadoop.hbase.util.VersionInfo: > Compiled by jenkins on Tue May 1 21:43:54 UTC 2012 > 2012-09-18 11:50:23,395 INFO > org.apache.zookeeper.server.ZooKeeperServer: Server > environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 > GMT > > ........ > > 2012-09-18 11:50:56,671 DEBUG > org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for > region -ROOT-,,0.70236052 > 2012-09-18 11:50:56,671 DEBUG > org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush > for -ROOT-,,0.70236052, current region memstore size 360.0 > 2012-09-18 11:50:56,671 DEBUG > org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting > -ROOT-,,0.70236052, commencing wait for mvcc, flushsize=360 > 2012-09-18 11:50:56,671 DEBUG > org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, > commencing flushing stores > 2012-09-18 11:50:56,684 DEBUG org.apache.hadoop.hbase.util.FSUtils: > Creating > file:hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13with > permission:rwxrwxrwx > 2012-09-18 11:50:56,692 DEBUG > org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with > CacheConfig:enabled [cacheDataOnRead=false] [cacheDataOnWrite=false] > [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] > [cacheEvictOnClose=false] [cacheCompressed=false] > 2012-09-18 11:50:56,694 INFO > org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom > filter type for > hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13: > CompoundBloomFilterWriter > 2012-09-18 11:50:56,703 INFO > org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and > NO DeleteFamily was added to HFile > (hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13) > 2012-09-18 11:50:56,703 INFO > org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=2, > memsize=360.0, into tmp file > hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 > 2012-09-18 11:50:56,716 WARN org.apache.hadoop.hdfs.DFSClient: > Exception while reading from blk_8430779885801230139_1008 of > /hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 from > 127.0.0.1:50010: java.io.IOException: BlockReader: error in packet > header(chunkOffset : 512, dataLen : 0, seqno : 0 (last: 0)) > at > org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1577) > at > org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237) > at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176) > at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193) > at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158) > > > Since my colleges can follow the same setup instruction and install it > in another machine (non-mac) I think this might be an issue with my > Macbook Pro? > > One thing I am not sure is if the system settings (max open files / > max user proc) needs to be adjusted. I've increased the max open files > # to 65536 already (as you can see from the beginning of the log). > > > The other thing I am not sure is why/how the file > hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 > is created. After failure to start HMaster, I checked that file with > dfs cat and get the same error: > > $ ./bin/hadoop dfs -cat > hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 > Warning: $HADOOP_HOME is deprecated. > 12/09/18 12:01:59 WARN hdfs.DFSClient: Exception while reading from > blk_8430779885801230139_1008 of > /hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 from > 127.0.0.1:50010: java.io.IOException: BlockReader: error in packet > header(chunkOffset : 512, dataLen : 0, seqno : 0 (last: 0)) > at > org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1577) > > > And this file definitely exists: > ./bin/hadoop dfs -ls hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/ > Warning: $HADOOP_HOME is deprecated. > Found 1 items > -rw-r--r-- 1 jasonhuang supergroup 848 2012-09-18 11:50 > /hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 > > > Also, when I look at some other dfs files they seem to be OK: > ./bin/hadoop dfs -cat > hdfs://localhost:54310/hbase/-ROOT-/70236052/.regioninfo > Warning: $HADOOP_HOME is deprecated. > > -ROOT-,,0-ROOT-?Y?? > > {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,} > > > > $ ./bin/hadoop dfs -cat > hdfs://localhost:54310/hbase/-ROOT-/70236052/.logs/hlog.1347983456546 > Warning: $HADOOP_HOME is deprecated. > SEQ0org.apache.hadoop.hbase.regionserver.wal.HLogKey0org.apache.hadoop.hbase.regionserver.wal.WALEditversion1g둣?????%???bV?"70236052-ROOT-9?9?????M#" > .META.,,1inforegioninfo9?9? .META.,,1.META.+???$ .META.,,1infov9?9? > > > Sorry for the lengthy email. Any help will be greatly appreciated! > > Jason > > On Thu, Sep 13, 2012 at 6:42 PM, Jason Huang <[email protected]> wrote: >> Hello, >> >> I am trying to set up HBase at pseudo-distributed mode on my Macbook. >> I was able to installed hadoop and HBase and started the nodes. >> >> $JPS >> 5417 TaskTracker >> 5083 NameNode >> 5761 HRegionServer >> 5658 HMaster >> 6015 Jps >> 5613 HQuorumPeer >> 5171 DataNode >> 5327 JobTracker >> 5262 SecondaryNameNode >> >> However, when I tried ./hbase shell I got the following error: >> Trace/BPT trap: 5 >>
