Re: read test with PE
Thanks Ted for looking into this.
I think the presplit opiton is the problem.
In step 1, I created the table with --presplit=3
In step 2, I run the read test and passed --presplit option as well.
As a result, the deletion happened during step 2.
Removing --presplit option at step 2 solved the problem.
Thanh
On Sun, Jun 2, 2013 at 11:02 AM, Ted Yu wrote:
> Looking at the tip of 0.94:
>
> private boolean checkTable(HBaseAdmin admin) throws IOException {
> HTableDescriptor tableDescriptor = getTableDescriptor();
> if (this.presplitRegions > 0) {
> // presplit requested
> if (admin.tableExists(tableDescriptor.getName())) {
> admin.disableTable(tableDescriptor.getName());
> admin.deleteTable(tableDescriptor.getName());
> }
> This means that if you specified --presplit=XX option in command line,
> pre-existing table would be dropped.
>
> Looking at the rev history
> for src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java, there
> was no change since 2012-02-27 (branch of 0.94)
>
> Can you check whether table deletion really happened during step 2 ?
>
> Cheers
>
> On Sun, Jun 2, 2013 at 8:55 AM, Thanh Do wrote:
>
>> and my hbase version is 0.94.2
>>
>>
>> On Sun, Jun 2, 2013 at 10:51 AM, Thanh Do wrote:
>>
>>> hi,
>>>
>>> I tried to run read test with PerformanceEvaluation (PE).
>>> What I did was.
>>> 1) create TestTable with a bunch of key-value first using PE
>>> sequentialWrite
>>> 2) run PE randomRead using following command.
>>> hbase org.apache.hadoop.hbase.PerformanceEvaluation randomRead 3
>>>
>>> However, from the log, it seems that at step 2, PE actually disables,
>>> deletes
>>> and recreates the TestTable before running the randomRead test.
>>> This means that the read test runs on an empty table!
>>>
>>> Is this an expected behavior, or did I do something wrong?
>>>
>>> Thanks,
>>> Thanh
>>>
>>>
>>>
>>>
>>
>
Re: read test with PE
and my hbase version is 0.94.2 On Sun, Jun 2, 2013 at 10:51 AM, Thanh Do wrote: > hi, > > I tried to run read test with PerformanceEvaluation (PE). > What I did was. > 1) create TestTable with a bunch of key-value first using PE > sequentialWrite > 2) run PE randomRead using following command. > hbase org.apache.hadoop.hbase.PerformanceEvaluation randomRead 3 > > However, from the log, it seems that at step 2, PE actually disables, > deletes > and recreates the TestTable before running the randomRead test. > This means that the read test runs on an empty table! > > Is this an expected behavior, or did I do something wrong? > > Thanks, > Thanh > > > >
read test with PE
hi, I tried to run read test with PerformanceEvaluation (PE). What I did was. 1) create TestTable with a bunch of key-value first using PE sequentialWrite 2) run PE randomRead using following command. hbase org.apache.hadoop.hbase.PerformanceEvaluation randomRead 3 However, from the log, it seems that at step 2, PE actually disables, deletes and recreates the TestTable before running the randomRead test. This means that the read test runs on an empty table! Is this an expected behavior, or did I do something wrong? Thanks, Thanh
Re: why multiple checkpoint nodes?
Thanks guys for updating! Yeah, I read the thread that Checkpoint/BackupNode may be get deprecated. SNN is a way to go then. I just wonder if we use multiple CheckpointNodes, we might run into the situation where while a checkpoint is on-going, but the first CheckpointNode is slow, then the second checkpointNode kicks in, just wonder what would happen. On Thu, Apr 18, 2013 at 3:07 PM, Mohammad Tariq wrote: > Hello Thanh, > >Just to keep you updated, checkpoint node might get depricated. So, > it's always better to use secondary namenode. More on this could be found > here : > https://issues.apache.org/jira/browse/HDFS-2397 > https://issues.apache.org/jira/browse/HDFS-4114 > > Warm Regards, > Tariq > https://mtariq.jux.com/ > cloudfront.blogspot.com > > > On Fri, Apr 19, 2013 at 1:15 AM, Bertrand Dechoux wrote: > >> It would be important to point the document (which I believe is >> http://hadoop.apache.org/docs/stable/hdfs_user_guide.html) and the >> version of Hadoop you are interested in. At one time, the documentation was >> misleading. The 1.x version didn't have checkpoint/backup nodes only the >> secondary namenode. I don't believe it has changed but I might be wrong (or >> the documentation still hasn't been fixed). The 2.x version will have >> namenode HA which will be the final solution. >> >> Regards >> >> Bertrand >> >> >> On Thu, Apr 18, 2013 at 7:20 PM, Thanh Do wrote: >> >>> so reliability (to prevent metadata loss) is the main motivation for >>> multiple checkpoint nodes? >>> >>> Does anybody use multiple checkpoint nodes in real life? >>> >>> Thanks >>> >>> >>> On Thu, Apr 18, 2013 at 12:07 PM, shashwat shriparv < >>> dwivedishash...@gmail.com> wrote: >>> >>>> more checkpoint nodes means more backup of the metadata :) >>>> >>>> *Thanks & Regards* >>>> >>>> ∞ >>>> Shashwat Shriparv >>>> >>>> >>>> >>>> On Thu, Apr 18, 2013 at 9:35 PM, Thanh Do wrote: >>>> >>>>> Hi all, >>>>> >>>>> The document says "Multiple checkpoint nodes may be specified in the >>>>> cluster configuration file". >>>>> >>>>> Can some one clarify me that why we really need to run multiple >>>>> checkpoint nodes anyway? Is it possible that while checkpoint node A is >>>>> doing checkpoint, and check point node B kicks in and does another >>>>> checkpoint? >>>>> >>>>> Thanks, >>>>> Thanh >>>>> >>>> >>>> >>> >> >> >> -- >> Bertrand Dechoux >> > >
Re: why multiple checkpoint nodes?
so reliability (to prevent metadata loss) is the main motivation for multiple checkpoint nodes? Does anybody use multiple checkpoint nodes in real life? Thanks On Thu, Apr 18, 2013 at 12:07 PM, shashwat shriparv < dwivedishash...@gmail.com> wrote: > more checkpoint nodes means more backup of the metadata :) > > *Thanks & Regards* > > ∞ > Shashwat Shriparv > > > > On Thu, Apr 18, 2013 at 9:35 PM, Thanh Do wrote: > >> Hi all, >> >> The document says "Multiple checkpoint nodes may be specified in the >> cluster configuration file". >> >> Can some one clarify me that why we really need to run multiple >> checkpoint nodes anyway? Is it possible that while checkpoint node A is >> doing checkpoint, and check point node B kicks in and does another >> checkpoint? >> >> Thanks, >> Thanh >> > >
Re: setting hdfs balancer bandwidth doesn't work
What do you mean by "doesn't work"? On Thu, Apr 18, 2013 at 10:01 AM, zhoushuaifeng wrote: > ** > Hi, > I set the hdfs balance bandwidth from 1048576 to 104857600, but it doesn't > work, what's wrong? > Does anyone encounter the same problem? > Thanks a lot. > > > dfs.balance.bandwidthPerSec > 104857600 > > Specifies the maximum amount of bandwidth that each datanode > can utilize for the balancing purpose in term of > the number of bytes per second. > > > > -- > zhoushuaifeng >
why multiple checkpoint nodes?
Hi all, The document says "Multiple checkpoint nodes may be specified in the cluster configuration file". Can some one clarify me that why we really need to run multiple checkpoint nodes anyway? Is it possible that while checkpoint node A is doing checkpoint, and check point node B kicks in and does another checkpoint? Thanks, Thanh
Re: threads quota is exceeded question
Hadoop by default limit 5 concurrent threads per node for balancing purpose. That causes your problem. On Mon, Apr 15, 2013 at 10:24 PM, rauljin wrote: > ** > HI: >The hadoop cluster is running balance. > >And one datannode 172.16.80.72 is : > > Datanode :Not able to copy block -507744952197054725 to / > 172.16.80.73:51658 because threads quota is exceeded. > > > ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 172.16.80.72:50010, storageID=DS-1202844662-172.16 > .80.72-50010-1330656432004, infoPort=50075, ipcPort=50020):DataXceiver > java.io.IOException: Block blk_8443528692263789109_8159545 is not valid. > > at > org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:734) > > at > org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:722) > > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:92) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:172) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95) > at java.lang.Thread.run(Thread.java:636) > > >And other datanode: > > > > ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration( > 172.16.80.73:50010 > , storageID=DS-1771394657-172.16.80.73-50010-1362474580654, infoPort=50075, > ipcPort=50020):DataXceiver > java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:267) > > at > org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:84) > > at > org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:92) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.replaceBlock(DataXceiver.java:580) > > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:115) > at java.lang.Thread.run(Thread.java:636) > > At that moment , the hdfs is not avalible. > > > I restart the 172.16.80.72 datanode service ,and the service is ok. > > > > what causes the problem. > > Any ideas? >Thanks! > > > > > > > > > > > -- > rauljin >
SequenceFile syncFs behavior?
Hi all, Could somebody clarify me the behavior of SequenceFile.syncFs(). From the source, I saw this function is deprecated, and hsync() or hflush() is recommended? However, it seems like current stable release of HBase (0.94) or the cloudera distribution (4.0) uses syncFs for its HLog file. Would syncFs() guarantee data durability? I am using hadoop 2.0. Many thanks, Thanh Do
