I can see the bug in the code snippet you provided! :) Basically, there's
an optimization where if the size of the compressed buffer is not
significantly less than the uncompressed buffer, then the block compressor
will store the uncompressed data as-is (e.g. compression_type == NONE). It
looks like there is a bug inside the BMZ decompression logic where it is not
properly handling this case. Luke wrote the BMZ compressor, so he'd be the
best person to take care of this one.
- Doug
On Wed, Aug 27, 2008 at 7:33 PM, Joshua Taylor <[EMAIL PROTECTED]>wrote:
> Is the BMZ compressor supported? The RangeServer crashed in the BMZ codec
> when I was dumping 500k random cells into a table using COMPRESSOR="bmz":
>
> Core was generated by
> `/data1/home/cosmix/hypertable/bin/Hypertable.RangeServer
> --pidfile=/data1/home/'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x0000000000608723 in Hypertable::BlockCompressionCodecBmz::inflate
> (this=0x174db00, [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED])
> at
> /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc:114
> 114
> /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc: No
> such file or directory.
> in
> /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc
> (gdb) directory hypertable/src/cc/Hypertable/Lib
> Source directories searched:
> /data1/home/cosmix/hypertable/src/cc/Hypertable/Lib:$cdir:$cwd
> (gdb) list
> 109 Error::BLOCK_COMPRESSOR_CHECKSUM_MISMATCH);
> 110
> 111 size_t outlen = header.get_data_length();
> 112
> 113 if (header.get_compression_type() == NONE)
> 114 memcpy(output.base, ip, outlen);
> 115 else {
> 116 output.reserve(outlen);
> 117 m_workmem.reserve(bmz_unpack_worklen(outlen), true);
> 118
> (gdb) print output
> $1 = (Hypertable::DynamicBuffer &) @0x1e0aac0: {base = 0x0, ptr = 0x0, size
> = 0, own = true}
> (gdb) print ip
> $2 = (const uint8_t *) 0xbf281a ""
> (gdb) print outlen
> $3 = 76
> (gdb) bt
> #0 0x0000000000608723 in Hypertable::BlockCompressionCodecBmz::inflate
> (this=0x174db00, [EMAIL PROTECTED], [EMAIL PROTECTED], [EMAIL PROTECTED])
> at
> /home/josh/hypertable/src/cc/Hypertable/Lib/BlockCompressionCodecBmz.cc:114
> #1 0x000000000059f859 in Hypertable::CellStoreV0::load_index
> (this=0x1e0aa00) at
> /home/josh/hypertable/src/cc/Hypertable/RangeServer/CellStoreV0.cc:474
> #2 0x000000000058b0f0 in Hypertable::AccessGroup::shrink (this=0xeefa00,
> [EMAIL PROTECTED]) at
> /home/josh/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:489
> #3 0x000000000057911f in Hypertable::Range::split_compact_and_shrink
> (this=0xeffc80) at
> /home/josh/hypertable/src/cc/Hypertable/RangeServer/Range.cc:563
> #4 0x000000000057ad95 in Hypertable::Range::split (this=0xeffc80) at
> /home/josh/hypertable/src/cc/Hypertable/RangeServer/Range.cc:341
> #5 0x0000000000576aad in Hypertable::MaintenanceTaskSplit::execute
> (this=0x1c2e9c0) at
> /home/josh/hypertable/src/cc/Hypertable/RangeServer/MaintenanceTaskSplit.cc:40
> #6 0x0000000000561d0a in Hypertable::MaintenanceQueue::Worker::operator()
> (this=0x4f017158) at
> /home/josh/hypertable/src/cc/Hypertable/RangeServer/MaintenanceQueue.h:108
> #7 0x0000000000561e73 in
> boost::detail::function::void_function_obj_invoker0<Hypertable::MaintenanceQueue::Worker,
> void>::invoke ([EMAIL PROTECTED])
> at
> /home/josh/hypertable/src/cc/boost-1_34-fix/boost/function/function_template.hpp:158
> #8 0x00002aaaab62551e in boost::function0<void,
> std::allocator<boost::function_base> >::operator() () from
> /usr/lib64/libboost_thread-mt.so.3
> #9 0x00002aaaab624f32 in boost::thread_group::join_all () from
> /usr/lib64/libboost_thread-mt.so.3
> #10 0x0000003986106337 in start_thread () from /lib64/libpthread.so.0
> #11 0x0000003790bcc38d in clone () from /lib64/libc.so.6
> #12 0x0000000000000000 in ?? ()
> (gdb)
>
> Looks like a memcpy to a null output buffer. Here's the tail of the
> RangeServer log:
>
> 1219889447 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:700)
> Successfully fetched 65453 bytes of scan data
> RangeServer::fetch_scanblock
> Scanner ID = 1
> 1219889447 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:700)
> Successfully fetched 65453 bytes of scan data
> RangeServer::fetch_scanblock
> Scanner ID = 1
> 1219889447 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:700)
> Successfully fetched 46970 bytes of scan data
> 1219889447 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:656)
> destroying scanner id=1
> 1219889447 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:177)
> Event: type=DISCONNECT from=10.10.1.162:43904
> 1219889453 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1883)
> Cleaning log (threshold=200000000)
> 1219889456 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/AsyncComm/IOHandler.h:81) Event:
> type=CONNECTION_ESTABLISHED from=10.10.1.162:43909
> 1219889456 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/ConnectionHandler.cc:167)
> Event: type=CONNECTION_ESTABLISHED from=10.10.1.162:43909
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (0 split off) updates to 'perf'
> 1219889457 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=4123295 items=67595 vm-est12234695
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (0 split off) updates to 'perf'
> 1219889457 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=5123329 items=83989 vm-est15202009
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (0 split off) updates to 'perf'
> 1219889457 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=6123363 items=100383 vm-est18169323
> 1219889457 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:79) RollLimit =
> 100000000
> 1219889457 DEBUG Hypertable.RangeServer : write
> (/home/josh/hypertable/src/cc/Hypertable/Lib/MetaLogDfsBase.cc:117):
> checksum=12916318 timestamp=1219889457515443000 type=1 payload=149
> 1219889457 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:288)
> Starting Major Compaction of perf[..<FF><FF>](default)
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (8518 split off) updates to 'perf'
> 1219889457 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=7123397 items=116777 vm-est21136637
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889457 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (9950 split off) updates to 'perf'
> 1219889457 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=8123431 items=133171 vm-est24103951
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (8246 split off) updates to 'perf'
> 1219889458 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=9123465 items=149565 vm-est27071265
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (7358 split off) updates to 'perf'
> 1219889458 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=10123499 items=165959 vm-est30038579
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (10666 split off) updates to 'perf'
> 1219889458 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=11123533 items=182353 vm-est33005893
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889458 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (7086 split off) updates to 'perf'
> 1219889458 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=12123567 items=198747 vm-est35973207
> RangeServer::update
> {TableIdentifier: name='perf' id='19' generation='1'}1219889459 INFO
> Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/AccessGroup.cc:430)
> Finished Compaction of perf[..<FF><FF>](default)
> 1219889459 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1188)
> Added 16394 (7826 split off) updates to 'perf'
> 1219889459 INFO Hypertable.RangeServer :
> (/home/josh/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:1286)
> drj mem=7000238 items=114758 vm-est20771198
> RangeServer::update
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
