Oh. Yeah. Makes sense. We used to bundle the native libs but we seem to have dropped them. We should add them back? St.Ack
On Tue, Jan 11, 2011 at 3:24 PM, Matt Corgan <[email protected]> wrote: > Turns out this is what happens if you don't have the native libraries set up > correctly. The data still gets compressed using the pure java codec, but it > doesn't cache the codec and gives you a warning each time it creates it for > each block. > > > On Mon, Jan 10, 2011 at 2:41 PM, Stack <[email protected]> wrote: > >> Thats a little silly. That the message is INFO level is probably >> small potatoes when doing a mapreduce job but in our case with lots of >> file openings, it turns into a little log storm. >> >> I suppose you'll need to disable it. Set log level to WARN on >> org.apache.hadoop.io.compress? >> >> This might help you making the change: >> http://wiki.apache.org/hadoop/Hbase/FAQ#A5 >> >> St.Ack >> >> On Mon, Jan 10, 2011 at 9:46 AM, Matt Corgan <[email protected]> wrote: >> > I'm trying to use GZIP compression but running into a logging problem. >> It >> > appears that every time a block is compressed it logs the following: >> > >> > 2011-01-10 12:40:48,407 INFO org.apache.hadoop.io.compress.CodecPool: Got >> > brand-new compressor >> > 2011-01-10 12:40:48,414 INFO org.apache.hadoop.io.compress.CodecPool: Got >> > brand-new compressor >> > 2011-01-10 12:40:48,420 INFO org.apache.hadoop.io.compress.CodecPool: Got >> > brand-new compressor >> > 2011-01-10 12:40:48,426 INFO org.apache.hadoop.io.compress.CodecPool: Got >> > brand-new compressor >> > 2011-01-10 12:40:48,431 INFO org.apache.hadoop.io.compress.CodecPool: Got >> > brand-new compressor >> > 2011-01-10 12:40:48,447 INFO org.apache.hadoop.io.compress.CodecPool: Got >> > brand-new compressor >> > 2011-01-10 12:40:48,453 INFO org.apache.hadoop.io.compress.CodecPool: Got >> > brand-new compressor >> > >> > Same for decompression. It's logging that 150 times per second during a >> > major compaction which pretty much renders the logs useless. I assume >> other >> > people are not having this problem, so did we accidentally enable that >> > logging somehow? >> > >> > Thanks, >> > Matt >> > >> >
