Data loss is still a problem untill Hadoop 0.21 (due in less than 3 months I heard) so it's always better to play it safer, but it has a price on performance.
If you really want to avoid losing data, I suggest taking a look at Hadoop branch 0.21 and the current HBase trunk. Both are relatively stable at the moment. J-D On Wed, Nov 11, 2009 at 2:07 AM, Peter Rietzler <[email protected]> wrote: > > Hi Jean > > I've managed to solve this issue without changing the configuration > properties. It seems that I've had another problem ... sorry. > > Puts are now ~ the same as in 0.19.3. Should that have been improved with > 0.20.1 ? > > I have not changed the hbase.regionserver.flushlogentries (this is set to > 100 in hbase-default.xml shipped with 0.20.1). Did you mean that I should > change this back to 100000. What implications would such a change have ? We > had problems with data loss in previous versions - does this change affect > this issue or is data loss not a problem with 0.20.1 anymore ? > > Thanks! > > Peter > > > > Jean-Daniel Cryans-2 wrote: >> >> Peter, >> >> One thing that changed is the default number of entries we keep in the >> write ahead logs that went from 100 000 to 100 mainly because we >> wanted to be safe in the case of 1MB rows. This can easily slow down >> the inserts a lot. To be sure, try resetting >> hbase.regionserver.flushlogentries to 100 in hbase-site.xml and >> restart HBase. >> >> J-D >> >> On Mon, Nov 9, 2009 at 10:23 AM, Peter Rietzler >> <[email protected]> wrote: >>> >>> Hi >>> >>> We currently switch from HBase 0.19.3 to 0.20.1 and experience very slow >>> inserts in comparison to 0.19.3. >>> >>> Inserting 10.000 rows with around 100 columns (pretty small content) per >>> row >>> takes around 26 seconds. I am using HTable.put(List<Row>) with batches of >>> 1000 rows. I've already set HTable.setAutoCommit(false) and tried to set >>> the >>> write buffer size to around 12MB (as seen in some previous posts). >>> >>> Our test currently just uses a standalone instance of HBase. With 0.19.3 >>> the >>> insert took a couple of seconds (using BatchUpdate). >>> >>> Can someone tell me what I'm doing wrong ? >>> >>> Thanks in advance! >>> >>> Cheers, >>> Peter >>> -- >>> View this message in context: >>> http://old.nabble.com/HTable.put%28%29-with-Hbase-0.20.1-tp26271241p26271241.html >>> Sent from the HBase User mailing list archive at Nabble.com. >>> >>> >> >> > > -- > View this message in context: > http://old.nabble.com/HTable.put%28%29-with-Hbase-0.20.1-tp26271241p26298872.html > Sent from the HBase User mailing list archive at Nabble.com. > >
