Hi J-D, I am using hadoop 1.0.3 - I was using dfs.datanode.data.dir last week but that had already been updated (someone else pointed that out) before I ran this test today.
thanks, Jason On Tue, Sep 18, 2012 at 1:05 PM, Jean-Daniel Cryans <[email protected]> wrote: > Which Hadoop version are you using exactly? I see you are setting > dfs.datanode.data.dir which is a post 1.0 setting (from what I can > tell by googling, since I didn't recognize it), but you are using a > "hadoop-examples-1.0.3.jar" file that seems to imply you are on 1.0.3 > which would probably not pick up dfs.datanode.data.dir > > J-D > > On Tue, Sep 18, 2012 at 9:21 AM, Jason Huang <[email protected]> wrote: >> I've done some more research but still can't start the HMaster node >> (with similar error). Here is what I found in the Master Server log: >> >> Tue Sep 18 11:50:22 EDT 2012 Starting master on Jasons-MacBook-Pro.local >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> file size (blocks, -f) unlimited >> max locked memory (kbytes, -l) unlimited >> max memory size (kbytes, -m) unlimited >> open files (-n) 65536 >> pipe size (512 bytes, -p) 1 >> stack size (kbytes, -s) 8192 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 1064 >> virtual memory (kbytes, -v) unlimited >> >> >> 2012-09-18 11:50:23,306 INFO org.apache.hadoop.hbase.util.VersionInfo: >> HBase 0.94.0 >> 2012-09-18 11:50:23,306 INFO org.apache.hadoop.hbase.util.VersionInfo: >> Subversion https://svn.apache.org/repos/asf/hbase/branches/0.94 -r >> 1332822 >> 2012-09-18 11:50:23,306 INFO org.apache.hadoop.hbase.util.VersionInfo: >> Compiled by jenkins on Tue May 1 21:43:54 UTC 2012 >> 2012-09-18 11:50:23,395 INFO >> org.apache.zookeeper.server.ZooKeeperServer: Server >> environment:zookeeper.version=3.4.3-1240972, built on 02/06/2012 10:48 >> GMT >> >> ........ >> >> 2012-09-18 11:50:56,671 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for >> region -ROOT-,,0.70236052 >> 2012-09-18 11:50:56,671 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush >> for -ROOT-,,0.70236052, current region memstore size 360.0 >> 2012-09-18 11:50:56,671 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting >> -ROOT-,,0.70236052, commencing wait for mvcc, flushsize=360 >> 2012-09-18 11:50:56,671 DEBUG >> org.apache.hadoop.hbase.regionserver.HRegion: Finished snapshotting, >> commencing flushing stores >> 2012-09-18 11:50:56,684 DEBUG org.apache.hadoop.hbase.util.FSUtils: >> Creating >> file:hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13with >> permission:rwxrwxrwx >> 2012-09-18 11:50:56,692 DEBUG >> org.apache.hadoop.hbase.io.hfile.HFileWriterV2: Initialized with >> CacheConfig:enabled [cacheDataOnRead=false] [cacheDataOnWrite=false] >> [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] >> [cacheEvictOnClose=false] [cacheCompressed=false] >> 2012-09-18 11:50:56,694 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: Delete Family Bloom >> filter type for >> hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13: >> CompoundBloomFilterWriter >> 2012-09-18 11:50:56,703 INFO >> org.apache.hadoop.hbase.regionserver.StoreFile: NO General Bloom and >> NO DeleteFamily was added to HFile >> (hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13) >> 2012-09-18 11:50:56,703 INFO >> org.apache.hadoop.hbase.regionserver.Store: Flushed , sequenceid=2, >> memsize=360.0, into tmp file >> hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 >> 2012-09-18 11:50:56,716 WARN org.apache.hadoop.hdfs.DFSClient: >> Exception while reading from blk_8430779885801230139_1008 of >> /hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 from >> 127.0.0.1:50010: java.io.IOException: BlockReader: error in packet >> header(chunkOffset : 512, dataLen : 0, seqno : 0 (last: 0)) >> at >> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1577) >> at >> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:237) >> at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:176) >> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:193) >> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158) >> >> >> Since my colleges can follow the same setup instruction and install it >> in another machine (non-mac) I think this might be an issue with my >> Macbook Pro? >> >> One thing I am not sure is if the system settings (max open files / >> max user proc) needs to be adjusted. I've increased the max open files >> # to 65536 already (as you can see from the beginning of the log). >> >> >> The other thing I am not sure is why/how the file >> hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 >> is created. After failure to start HMaster, I checked that file with >> dfs cat and get the same error: >> >> $ ./bin/hadoop dfs -cat >> hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 >> Warning: $HADOOP_HOME is deprecated. >> 12/09/18 12:01:59 WARN hdfs.DFSClient: Exception while reading from >> blk_8430779885801230139_1008 of >> /hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 from >> 127.0.0.1:50010: java.io.IOException: BlockReader: error in packet >> header(chunkOffset : 512, dataLen : 0, seqno : 0 (last: 0)) >> at >> org.apache.hadoop.hdfs.DFSClient$BlockReader.readChunk(DFSClient.java:1577) >> >> >> And this file definitely exists: >> ./bin/hadoop dfs -ls hdfs://localhost:54310/hbase/-ROOT-/70236052/.tmp/ >> Warning: $HADOOP_HOME is deprecated. >> Found 1 items >> -rw-r--r-- 1 jasonhuang supergroup 848 2012-09-18 11:50 >> /hbase/-ROOT-/70236052/.tmp/3c6caf495a1743eca405a5f59edaef13 >> >> >> Also, when I look at some other dfs files they seem to be OK: >> ./bin/hadoop dfs -cat >> hdfs://localhost:54310/hbase/-ROOT-/70236052/.regioninfo >> Warning: $HADOOP_HOME is deprecated. >> >> -ROOT-,,0-ROOT-?Y?? >> >> {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', ENCODED => 70236052,} >> >> >> >> $ ./bin/hadoop dfs -cat >> hdfs://localhost:54310/hbase/-ROOT-/70236052/.logs/hlog.1347983456546 >> Warning: $HADOOP_HOME is deprecated. >> SEQ0org.apache.hadoop.hbase.regionserver.wal.HLogKey0org.apache.hadoop.hbase.regionserver.wal.WALEditversion1g둣?????%???bV?"70236052-ROOT-9?9?????M#" >> .META.,,1inforegioninfo9?9? .META.,,1.META.+???$ .META.,,1infov9?9? >> >> >> Sorry for the lengthy email. Any help will be greatly appreciated! >> >> Jason >> >> On Thu, Sep 13, 2012 at 6:42 PM, Jason Huang <[email protected]> wrote: >>> Hello, >>> >>> I am trying to set up HBase at pseudo-distributed mode on my Macbook. >>> I was able to installed hadoop and HBase and started the nodes. >>> >>> $JPS >>> 5417 TaskTracker >>> 5083 NameNode >>> 5761 HRegionServer >>> 5658 HMaster >>> 6015 Jps >>> 5613 HQuorumPeer >>> 5171 DataNode >>> 5327 JobTracker >>> 5262 SecondaryNameNode >>> >>> However, when I tried ./hbase shell I got the following error: >>> Trace/BPT trap: 5 >>>
