Thank for the relise!
I installed it.
Have a crash :(
>the error occurs on the same RangeServer every time?
sorry, dunno
RangeServer Logs:
...
1344251972 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
Memory Usage: 2317990351 bytes
1344251981 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83)
QueryCache hit rate over last 1000 lookups, cumulative = 11.000000,
16.383754
1344251984 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3038)
Entering get_statistics()
1344251984 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3283)
Exiting get_statistics()
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RSStats.h:83)
Maintenance stats scans=(2811 1011 2972785 0.074320) updates=(317 1915
12652963 0.316324 312)
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
fragments from '/hypertable/servers/rs1/log/root' with latest revision
older than 1312045326
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
fragments from '/hypertable/servers/rs1/log/metadata' with latest revision
older than 131204
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
fragments from '/hypertable/servers/rs1/log/system' with latest revision
older than 13120524
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/Lib/CommitLog.cc:319) Purging log
fragments from '/hypertable/servers/rs1/log/user' with latest revision
older than 1343832886
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:271)
Memory Statistics (MB): VM=3065.34, RSS=2373.27, tracked=2218.39,
computed=2218.56 limit=3244.80
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/MaintenanceScheduler.cc:276)
Memory Allocation: BlockCache=46.67% BlockIndex=0.19% BloomFilter=0.18%
CellCache=50.81% ShadowCache=0.00% QueryCache=2.15%
1344251992 INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/RangeServer.cc:3777)
Memory Usage: 2326151392 bytes
*1344251993* INFO Hypertable.RangeServer :
(/root/src/hypertable/src/cc/Hypertable/RangeServer/QueryCache.cc:83)
QueryCache hit rate over last 1000 lookups, cumulative = 11.100000,
16.375433
*terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called after throwing an instance of 'std::bad_alloc'*
[[EOF]]
ThriftBroker Logs just after crash:
...
*1344252038* ERROR ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket
read(29, len=38) failure : Connection reset by peer
1344252038 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756)
FileUtils::writev(29, len=252) failed : Broken pipe
1344252039 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem
flushing send queue - COMM broken connection
1344252039 INFO ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/ConnectionManager.cc:359) Event:
type=DISCONNECT from=178.162.111.111:38060; Problem connecting to Root
RangeServer, will retry in 3000 milliseconds...
1344252039 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:756)
FileUtils::writev(29, len=252) failed : Bad file descriptor
1344252039 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:703) Problem
flushing send queue - COMM broken connection
1344252039 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252039 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252039 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252039 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
...
(when I try to get data):
...
1344252654 ERROR ThriftBroker : TThreadedServer: Caught TException:
pthread_create failed
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
1344252654 ERROR ThriftBroker : dump_error_history
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:144):
Hypertable::Exception: Problem creating scanner for start row 'some_data'
on METADATA[..??] - COMM not connected
at int Hypertable::RangeLocator::find(const
Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*,
Hypertable::Timer&, bool)
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:352)
at void Hypertable::RangeServerClsrc/central_freelist.cc:322]
tcmalloc: allocation failed 16384
1344252654 ERROR ThriftBroker : dump_error_history
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.h:144):
Hypertable::Exception: Problem creating scanner for start row 'some_data'
on METADATA[..??] - COMM not connected
at int Hypertable::RangeLocator::find(const
Hypertable::TableIdentifier*, const char*, Hypertable::RangeLocationInfo*,
Hypertable::Timer&, bool)
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeLocator.cc:352)
at void Hypertable::RangeServesrc/central_freelist.cc:322]
tcmalloc: allocation failed 16384
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
1344252654 ERROR ThriftBroker : get_cells
(/root/src/hypertable/src/cc/ThriftBroker/ThriftBroker.cc:909): virtual
void
Hypertable::ThriftBroker::ServerHandler::get_cells(Hypertable::ThriftBroker::ThriftCells&,
Hypertable::ThriftGen::Namespace, const Hypertable::String&, const
Hypertable::ThriftGen::ScanSpec&)namespace=38465 table=SomeTable
scan_spec={ScanSpec: cells=[
{CellInterval: start_row='some_data' start_column='data'
start_inclusive=1 end_row='some_data' src/central_freelist.cc:322]
tcmalloc: allocation failed 8192
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
...
(in little time):
...
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252654 WARN ThriftBroker :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
src/central_freelist.cc:322] tcmalloc: allocation failed 16384
src/central_freelist.cc:322] tcmalloc: allocation failed 8192
*terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc*
[[EOF]]
Master Logs (before and ater crash):
sh: dot: not found
1344251984 ERROR Hypertable.Master :
(/root/src/hypertable/src/cc/Common/FileUtils.cc:451)
rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg",
"/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
ERROR:
/opt/hypertable/0.9.6.0/run/monitoring/rangeservers/rs1_stats_v0.rrd: found
extra data on update argument: 7.50:1.44
1344251984 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/Monitoring.cc:786) Monitor:
failed to invoke `rrdtool`; make sure it's properly installed and in your
$PATH (command returne
1344251984 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
Leaving GatherStatistics-450
*1344252014* INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
Entering GatherStatistics-452 state=INITIAL
1344252014 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53)
Entering LoadBalancer-453
1344252014 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72)
Leaving LoadBalancer-453
sh: dot: not found
1344252014 ERROR Hypertable.Master :
(/root/src/hypertable/src/cc/Common/FileUtils.cc:451)
rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg",
"/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") failed - No such file or
directory
1344252038 ERROR Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/IOHandlerData.cc:237) socket
read(27, len=38) failure : *Connection reset by peer*
1344252038 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
Leaving GatherStatistics-452
1344252038 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/ConnectionHandler.cc:195)
*Event:
type=DISCONNECT from=178.162.190.234:38938*
1344252044 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
Entering GatherStatistics-455 state=INITIAL
1344252044 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252044 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53)
Entering LoadBalancer-456
1344252044 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99) *Found
non-live server rs1 wait till all servers are live before trying balance*
1344252044 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72)
Leaving LoadBalancer-456
1344252044 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
*Comm::send_request
to rs1 failed - COMM not connected*
sh: dot: not found
1344252044 ERROR Hypertable.Master :
(/root/src/hypertable/src/cc/Common/FileUtils.cc:451)
rename("/opt/hypertable/0.9.6.0/run/monitoring/mop.tmp.jpg",
"/opt/hypertable/0.9.6.0/run/monitoring/mop.jpg") faile
1344252044 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:100)
Leaving GatherStatistics-455
1344252074 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationGatherStatistics.cc:57)
Entering GatherStatistics-457 state=INITIAL
1344252074 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/AsyncComm/Comm.cc:246) No connection for rs1 -
COMM not connected
1344252074 WARN Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Lib/RangeServerClient.cc:678)
Comm::send_request to rs1 failed - COMM not connected
1344252074 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:53)
Entering LoadBalancer-458
1344252074 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/LoadBalancerBasic.cc:99)
Found non-live server rs1 wait till all servers are live before trying
balance
1344252074 INFO Hypertable.Master :
(/root/src/hypertable/src/cc/Hypertable/Master/OperationLoadBalancer.cc:72)
Leaving LoadBalancer-458
sh: dot: not found
...
HyperSapce Logs (just after crach):
...
*1344252038* INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42)
Destroying session 3
1344252038 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session
3(Hypertable.RangeServer)
1344252039 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 3
name=Hypertable.RangeServer
1344252039 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 9
1344252039 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 10
1344252039 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1523) Persisting lock
released notifications
1344252039 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:1534) Finished persisting
lock released notifications
1344252039 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 11
1344252657 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/RequestHandlerDestroySession.cc:42)
Destroying session 4
1344252657 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:312) destroyed session
4(ThriftBroker)
1344252658 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:463) Expiring session 4
name=ThriftBroker
1344252658 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 14
1344252658 INFO Hyperspace.Master :
(/root/src/hypertable/src/cc/Hyperspace/Master.cc:482) Destroying handle 15
[[EOF]]
DfsBroker Logs (before and just after crash):
...
1344251305 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.cc:137) open(
/hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs18 ) = 367 (local=59)
1344251305 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
/hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs17 , 41 )
1344251305 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
/hypertable/tables/2/7/default/FDyEnEg1nT03Mrcj/cs17 , 49 )
1344251519 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
/hypertable/servers/rs1/log/user/264 , 39 )
1344251519 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.cc:200) create(
/hypertable/servers/rs1/log/user/265 ) = 368 (local=39)
*1344252038* INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/Lib/OpenFileMap.h:88) Removing
handle 4 from open file map because of lost owning client connection
1344252038 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
/hypertable/servers/rs1/log/rsml/52 , 26 )
1344252038 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/Lib/OpenFileMap.h:88) Removing
handle 6 from open file map because of lost owning client connection
1344252038 INFO localBroker :
(/root/src/hypertable/src/cc/DfsBroker/local/LocalBroker.h:49) close(
/hypertable/servers/rs1/log/root/1 , 27 )
...
On Sunday, August 5, 2012 5:12:10 PM UTC+3, Doug Judd wrote:
>
> Here are 32-bit versions of the 0.9.6.1 pre-release:
>
>
> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.a0551dd-linux-i386.deb
>
> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.a0551dd-linux-i386.tar.bz2
>
> It's a normal release build. Give this package a try and if the problem
> still persists, please let us know. One other question, do you know if the
> error occurs on the same RangeServer every time?
>
> - Doug
>
> On Sat, Aug 4, 2012 at 12:13 PM, Kenny F. (2) <[email protected]> wrote:
>
>> Thank you for help!
>> I work on Debian, *32* bit.
>>
>> P.S. ulimit -v is unlimited
>>
>> Doug Judd wrote:
>> > Hi Kenny,
>> >
>> >
>> > We've fixed a couple of bugs recently that may be the source of this
>> problem. Can you trying it out by installing the following pre-release of
>> 0.9.6.1 and re-running your test?
>> >
>> >
>> >
>> >
>> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.d051768-linux-x86_64.tar.bz2
>> >
>> >
>> >
>> > --
>> >
>> > Doug Judd
>> > CEO, Hypertable Inc.
>>
>>
>>
>> Doug Judd wrote:
>> > Hi Kenny,
>> >
>> >
>> > We've fixed a couple of bugs recently that may be the source of this
>> problem. Can you trying it out by installing the following pre-release of
>> 0.9.6.1 and re-running your test?
>> >
>> >
>> >
>> >
>> http://download.hypertable.com/pub/pre-releases/hypertable-0.9.6.0.d051768-linux-x86_64.tar.bz2
>> >
>> >
>> >
>> > But before you do that, please verify that you have no virtual memory
>> limitations in place:
>> >
>> >
>> > ulimit -v
>> > (or 'ulimit memorysize' in csh)
>> >
>> > - Doug
>>
>>>
>
>
> --
> Doug Judd
> CEO, Hypertable Inc.
>
>
--
You received this message because you are subscribed to the Google Groups
"Hypertable Development" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/hypertable-dev/-/Gxgo8LLcHlAJ.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/hypertable-dev?hl=en.
