Thank for the release!
I installed it.
Have a crash :(
>the error occurs on the same RangeServer every time?
sorry, dunno
RangeServer Logs:
...
1344251972 INFO Hypertable.RangeServer : (/root/src/hypertable/src/cc/
>
> Hypertable/RangeServer/RangeServer.cc:3777) Memory Usage: 2317990351 bytes
> 1344251981 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83)
> QueryCache hit rate over last 1000 lookups, cumulative = 11.000000,
> 16.383754
> 1344251984 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038)
> Entering get_statistics()
> 1344251984 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283)
> Exiting get_statistics()
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83)
> Maintenance stats scans=(2811 1011 2972785 0.074320) updates=(317 1915
> 12652963 0.316324 312)
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
> fragments from '/hypertable/servers/rs1/log/root' with latest revision
> older than 1312045326
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
> fragments from '/hypertable/servers/rs1/log/metadata' with latest revision
> older than 131204
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
> fragments from '/hypertable/servers/rs1/log/system' with latest revision
> older than 13120524
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
> fragments from '/hypertable/servers/rs1/log/user' with latest revision
> older than 1343832886
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271)
>
> Memory Statistics (MB): VM=3065.34, RSS=2373.27, tracked=2218.39,
> computed=2218.56 limit=3244.80
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
>
> Memory Allocation: BlockCache=46.67% BlockIndex=0.19% BloomFilter=0.18%
> CellCache=50.81% ShadowCache=0.00% QueryCache=2.15%
> 1344251992 INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
> Memory Usage: 2326151392 bytes
> *1344251993* INFO Hypertable.RangeServer :
> (/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83)
> QueryCache hit rate over last 1000 lookups, cumulative = 11.100000,
> 16.375433
> *terminate called recursively
> terminate called recursively
> terminate called recursively
> terminate called recursively
> terminate called after throwing an instance of 'std::bad_alloc'*
> [[EOF]]
>
> ThriftBroker Logs just after crash:
> ...
> *1344252038* ERROR ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket
> read(29, len=38) failure : Connection reset by peer
> 1344252038 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756)
> FileUtils::writev(29, len=252) failed : Broken pipe
> 1344252039 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem
> flushing send queue - COMM broken connection
> 1344252039 INFO ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event:
> type=DISCONNECT from=178.162.111.111:38060; Problem connecting to Root
> RangeServer, will retry in 3000 milliseconds...
> 1344252039 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756)
> FileUtils::writev(29, len=252) failed : Bad file descriptor
> 1344252039 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem
> flushing send queue - COMM broken connection
> 1344252039 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252039 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252039 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252039 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> ...
> (when I try to get data):
> ...
> 1344252654 ERROR ThriftBroker : TThreadedServer: Caught TException:
> pthread_create failed
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> Comm::send_request to rs1 failed - COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> Comm::send_request to rs1 failed - COMM not connected
> src/central_freelist.cc:322] tcmalloc: allocation failed 16384
> 1344252654 ERROR ThriftBroker : dump_error_history
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:144):
> Hypertable::Exception: Problem creating scanner for start row 'some_data'
> on METADATA[..??] - COMM not connected
> at int Hypertable::RangeLocator::find(const
> Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*,
> Hypertable::Timer&, bool)
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:352)
> at void Hypertable::RangeServerClsrc/central_freelist.cc:322]
> tcmalloc: allocation failed 16384
> 1344252654 ERROR ThriftBroker : dump_error_history
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:144):
> Hypertable::Exception: Problem creating scanner for start row 'some_data'
> on METADATA[..??] - COMM not connected
> at int Hypertable::RangeLocator::find(const
> Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*,
> Hypertable::Timer&, bool)
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:352)
> at void Hypertable::RangeServesrc/central_freelist.cc:322]
> tcmalloc: allocation failed 16384
> src/central_freelist.cc:322] tcmalloc: allocation failed 16384
> 1344252654 ERROR ThriftBroker : get_cells
> (/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:909): virtual
> void
> Hypertable::ThriftBroker::ServerHandler::get_cells(Hypertable::ThriftBroker::ThriftCells&,
>
> Hypertable::ThriftGen::Namespace, const Hypertable::String&, const
> Hypertable::ThriftGen::ScanSpec&)namespace=38465 table=SomeTable
> scan_spec={ScanSpec: cells=[
> {CellInterval: start_row='some_data' start_column='data'
> start_inclusive=1 end_row='some_data' src/central_freelist.cc:322]
> tcmalloc: allocation failed 8192
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> Comm::send_request to rs1 failed - COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> Comm::send_request to rs1 failed - COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> Comm::send_request to rs1 failed - COMM not connected
> ...
> (in little time):
> ...
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252654 WARN ThriftBroker :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> Comm::send_request to rs1 failed - COMM not connected
> src/central_freelist.cc:322] tcmalloc: allocation failed 16384
> src/central_freelist.cc:322] tcmalloc: allocation failed 8192
> *terminate called after throwing an instance of 'std::bad_alloc'
> what(): std::bad_alloc*
> [[EOF]]
>
>
> Master Logs (before and ater crash):
> sh: dot: not found
> 1344251984 ERROR Hypertable.Master :
> (/root/src/hypertable/src/cc/Common/FileUtils.cc:451)
> rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg",
> "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
> ERROR: /opt/hypertable/
> 0.9.6.0/run/monitoring/rangeservers/rs1_stats_v0.rrd: found extra data on
> update argument: 7.50:1.44
> 1344251984 WARN Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor:
> failed to invoke `rrdtool`; make sure it's properly installed and in your
> $PATH (command returne
> 1344251984 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
>
> Leaving GatherStatistics-450
> *1344252014* INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
>
> Entering GatherStatistics-452 state=INITIAL
> 1344252014 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53)
> Entering LoadBalancer-453
> 1344252014 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72)
> Leaving LoadBalancer-453
> sh: dot: not found
> 1344252014 ERROR Hypertable.Master :
> (/root/src/hypertable/src/cc/Common/FileUtils.cc:451)
> rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg",
> "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") failed - No such file
> or directory
> 1344252038 ERROR Hypertable.Master :
> (/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket
> read(27, len=38) failure : *Connection reset by peer*
> 1344252038 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
>
> Leaving GatherStatistics-452
> 1344252038 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:195)
> *Event:
> type=DISCONNECT from=178.162.111.111:38938 <http://178.162.190.234:38938>*
> 1344252044 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
>
> Entering GatherStatistics-455 state=INITIAL
> 1344252044 WARN Hypertable.Master :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252044 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53)
> Entering LoadBalancer-456
> 1344252044 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99)
> *Found
> non-live server rs1 wait till all servers are live before trying balance*
> 1344252044 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72)
> Leaving LoadBalancer-456
> 1344252044 WARN Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> *Comm::send_request
> to rs1 failed - COMM not connected*
> sh: dot: not found
> 1344252044 ERROR Hypertable.Master :
> (/root/src/hypertable/src/cc/Common/FileUtils.cc:451)
> rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg",
> "/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
> 1344252044 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
>
> Leaving GatherStatistics-455
> 1344252074 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
>
> Entering GatherStatistics-457 state=INITIAL
> 1344252074 WARN Hypertable.Master :
> (/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
> COMM not connected
> 1344252074 WARN Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
> Comm::send_request to rs1 failed - COMM not connected
> 1344252074 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53)
> Entering LoadBalancer-458
> 1344252074 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99)
> Found non-live server rs1 wait till all servers are live before trying
> balance
> 1344252074 INFO Hypertable.Master :
> (/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72)
> Leaving LoadBalancer-458
> sh: dot: not found
> ...
>
>
> HyperSapce Logs (just after crach):
> ...
> *1344252038* INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42)
> Destroying session 3
> 1344252038 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session
> 3(Hypertable.RangeServer)
> 1344252039 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 3
> name=Hypertable.RangeServer
> 1344252039 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 9
> 1344252039 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 10
> 1344252039 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1523) Persisting lock
> released notifications
> 1344252039 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:1534) Finished persisting
> lock released notifications
> 1344252039 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 11
> 1344252657 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42)
> Destroying session 4
> 1344252657 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session
> 4(ThriftBroker)
> 1344252658 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 4
> name=ThriftBroker
> 1344252658 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 14
> 1344252658 INFO Hyperspace.Master :
> (/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 15
> [[EOF]]
>
>
> DfsBroker Logs (before and just after crash):
> ...
> 1344251305 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.cc:137) open(
> /hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs18 ) = 367 (local=59)
> 1344251305 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
> /hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs17 , 41 )
> 1344251305 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
> /hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs17 , 49 )
> 1344251519 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
> /hypertable/servers/rs1/log/user/264 , 39 )
> 1344251519 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.cc:200) create(
> /hypertable/servers/rs1/log/user/265 ) = 368 (local=39)
> *1344252038* INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/Lib/OpenFileMap.h:88) Removing
> handle 4 from open file map because of lost owning client connection
> 1344252038 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
> /hypertable/servers/rs1/log/rsml/52 , 26 )
> 1344252038 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/Lib/OpenFileMap.h:88) Removing
> handle 6 from open file map because of lost owning client connection
> 1344252038 INFO localBroker :
> (/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
> /hypertable/servers/rs1/log/root/1 , 27 )
> ...
>
>
>
> On Sunday, August 5, 2012 5:12:10 PM UTC+3, Doug Judd wrote:
>>
>> Here are 32-bit versions of the 0.9.6.1 pre-release:
>>
>>
>> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.a0551dd-linux-i386.deb
>>
>> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.a0551dd-linux-i386.tar.bz2
>>
>> It's a normal release build. Give this package a try and if the problem
>> still persists, please let us know. One other question, do you know if the
>> error occurs on the same RangeServer every time?
>>
>> - Doug
>>
>> On Sat, Aug 4, 2012 at 12:13 PM, Kenny F. (2) <[email protected]> wrote:
>>
>>> Thank you for help!
>>> I work on Debian, *32* bit.
>>>
>>> P.S. ulimit -v is unlimited
>>>
>>> Doug Judd wrote:
>>> > Hi Kenny,
>>> >
>>> >
>>> > We've fixed a couple of bugs recently that may be the source of this
>>> problem. Can you trying it out by installing the following pre-release of
>>> 0.9.6.1 and re-running your test?
>>> >
>>> >
>>> >
>>> >
>>> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.d051768-linux-x86_64.tar.bz2
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Doug Judd
>>> > CEO, Hypertable Inc.
>>>
>>>
>>>
>>> Doug Judd wrote:
>>> > Hi Kenny,
>>> >
>>> >
>>> > We've fixed a couple of bugs recently that may be the source of this
>>> problem. Can you trying it out by installing the following pre-release of
>>> 0.9.6.1 and re-running your test?
>>> >
>>> >
>>> >
>>> >
>>> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.d051768-linux-x86_64.tar.bz2
>>> >
>>> >
>>> >
>>> > But before you do that, please verify that you have no virtual memory
>>> limitations in place:
>>> >
>>> >
>>> > ulimit -v
>>> > (or 'ulimit memorysize' in csh)
>>> >
>>> > - Doug
>>>
>>>>
>>
>>
>> --
>> Doug Judd
>> CEO, Hypertable Inc.
>>
>
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/hypertable-dev/-/lyg8hS4e0nIJ.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en.
