libgplcompression.so in hbase/lib/native/<arch> on all region server nodes?
hadoop-gpl-compression.jar in hbase/lib on all region server nodes? Run at DEBUG log level and initiate major_compaction. Anything relevant in the master log? Anything relevant in the region server logs? - Andy ________________________________ From: Igor Ranitovic <[email protected]> To: [email protected] Sent: Wednesday, September 2, 2009 2:52:11 PM Subject: Re: major_compaction and LZO I checked a HFile and it does not look compressed. Any ideas on how to proceed with troubleshooting? Thanks. i. > Regionserver logs will mention if major compaction happened on a region > or not, but maybe only at DEBUG log level, I'm not sure. I was running > with DEBUG at the time and saw immediately on a region server log tail > that the store files were rewritten with LZO. > > I downloaded a HFile out of a randomly chosen region via 'hadoop fs > -copyToLocal ...' and looked at it to confirm it was compressed. > >> hbase(main):013: major_compact 'urls' >> 0 row(s) in 0.0460 seconds > > 0 row result for an admin command doesn't mean anything. This tells you the > command was accepted. If there was a problem, an exception would be thrown. > > - Andy > > > > > ________________________________ > From: Igor Ranitovic <[email protected]> > To: [email protected] > Sent: Wednesday, September 2, 2009 2:12:37 PM > Subject: Re: major_compaction and LZO > > I tried again, but there is nothing in the regionserver logs. > The size of the table is tiny (MBs). > > In general, what would be a good way to verify if LZO compression is > working. > > i. > >> major compaction is not instant... I've seen it take an hour on a >> 300-400gb table. You want to check the regionserver logs looking for >> any major compaction lines. >> >> in your case, the table will take a lot less longer, but still not >> instant. Give it another shot, see what happens. >> >> -ryan >> >> On Wed, Sep 2, 2009 at 12:57 PM, Igor Ranitovic<[email protected]> wrote: >>> After setting up a test table with about 250K rows, I wanted to enable >>> lzo compression on it. >>> >>> In the shell, I disabled and altered the table and then ran >>> major_compaction. >>> >>> This is the result: >>> >>> hbase(main):013: major_compact 'urls' >>> 0 row(s) in 0.0460 seconds >>> >>> Does '0 rows' means that this did not work? >>> (the size of the table on the disk did not change/reduce) >>> >>> Table info: >>> hbase(main):019:0> describe 'urls' >>> DESCRIPTION >>> ENABLED >>> >>> false >>> {NAME => 'urls', FAMILIES => [{NAME => 'data', VERSIONS => '1', >>> COMPRE >>> SSION => 'LZO', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY >>> = >>> > 'false', BLOCKCACHE => 'true'}]} >>> 1 row(s) in 0.0420 seconds >>> >>> >>> Table Count: >>> hbase(main):021:0> count 'urls' >>> .... >>> Current count: 251000, row: http://com,zz/ >>> >>> >>> From the master log: >>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,880 INFO >>> com.hadoop.compression.lzo.GPLNativeCodeLoader: Loaded native gpl >>> library >>> hbase-crawler-master-xxx.log:2009-09-02 13:54:51,891 INFO >>> com.hadoop.compression.lzo.LzoCodec: Successfully loaded & initialized >>> native-lzo library >>> >>> I using 0.20. Thanks. >>> >>> i. >>> > > >
