Did you cut and paste this log transcript with multiple pastes? There's a really weird view repetition but I'm going to just assume that you accidentally overlapped a copy/paste into email for now.
On Tue, Jun 5, 2012 at 5:07 AM, Sami Sierla <[email protected]> wrote: > Dave, > > Thank You for quick reply. The issues appear in a production environment to > which I don't have access to modify configuration or design documents. Log > level at the moment is "error" > > Below is a lengthy log dump we got when the os_process_timeout was 5000, > after increasing timeout to 30000 there has been no log entries at all when > indexing stops. > > ----- > > [Thu, 31 May 2012 17:42:17 GMT] [error] [<0.15656.0>] OS Process Error > <0.15657.0> :: {os_process_error, "OS process timed out."} > [Thu, 31 May 2012 17:42:17 GMT] [error] [emulator] Error in process > <0.15656.0> with exit value: {{nocatch,{os_process_error,"OS process timed > out."}},[{couch_os_process,prompt,2},{couch_query_servers,map_doc_raw,2},{couch_view_updater,'-do_maps/3-fun-0-',3},{couch_view_updater,do_maps,3}]} > [Thu, 31 May 2012 17:42:17 GMT] [error] [<0.15648.0>] ** Generic server > <0.15648.0> terminating > ** Last message in was {'EXIT',<0.15653.0>, > {{nocatch, > {os_process_error,"OS process timed out."}}, > [{couch_os_process,prompt,2}, > {couch_query_servers,map_doc_raw,2}, > {couch_view_updater,'-do_maps/3-fun-0-',3}, > {couch_view_updater,do_maps,3}]}} > ** When Server state == {group_state,undefined,<<"mutka_replicated">>, > {"/data/mutka/couchdb-index",<<"mutka_replicated">>, > {group, > <<223,185,95,248,235,18,77,64,18,164,253,96,95,237, > 204,20>>, > nil,<<"_design/transactionA-1.2.0">>, > <<"javascript">>,[], > [{view,0,0,0, > [<<"transactionByPaymentInstrument">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.paymentInstrumentId) { > emit([doc.paymentInstrumentId,doc.startTimestamp], null); } }">>, > nil,[],[]}, > {view,1,0,0, > [<<"transactionByTerminal">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.paymentTerminalId) { > emit([doc.paymentTerminalId,doc.startTimestamp], null); } }">>, > nil,[],[]}, > {view,2,0,0, > [<<"transactionBySession">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.protocolSessionId) { > emit(doc.protocolSessionId,doc.protocolTransactionId); } }">>, > nil,[],[]}, > {view,3,0,0, > [<<"transactionByRayId">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.cId) { emit([-(-doc.cId),doc.startTimestamp], > null); } }">>, > nil,[],[]}], > {[]}, > nil,0,0,nil,nil}}, > {group, > <<223,185,95,248,235,18,77,64,18,164,253,96,95,237, > 204,20>>, > <0.15650.0>,<<"_design/transactionA-1.2.0">>, > <<"javascript">>,[], > [{view,0,236439939,0, > [<<"transactionByPaymentInstrument">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.paymentInstrumentId) { > emit([doc.paymentInstrumentId,doc.startTimestamp], null); } }">>, > {btree,<0.15650.0>, > {47573274456,{8694059,[]},257926106}, > #Fun<couch_btree.3.71804109>, > #Fun<couch_btree.4.115144917>, > #Fun<couch_view.less_json_ids.2>, > #Fun<couch_view_group.10.26766604>,snappy}, > [],[]}, > {view,1,236439939,0, > [<<"transactionByTerminal">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.paymentTerminalId) { > emit([doc.paymentTerminalId,doc.startTimestamp], null); } }">>, > {btree,<0.15650.0>, > {47574093427,{33638477,[]},942288018}, > #Fun<couch_btree.3.71804109>, > #Fun<couch_btree.4.115144917>, > #Fun<couch_view.less_json_ids.2>, > #Fun<couch_view_group.10.26766604>,snappy}, > [],[]}, > {view,2,236439939,0, > [<<"transactionBySession">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.protocolSessionId) { > emit(doc.protocolSessionId,doc.protocolTransactionId); } }">>, > {btree,<0.15650.0>, > {47574114746,{9241366,[]},131141244}, > #Fun<couch_btree.3.71804109>, > #Fun<couch_btree.4.115144917>, > #Fun<couch_view.less_json_ids.2>, > #Fun<couch_view_group.10.26766604>,snappy}, > [],[]}, > {view,1,236439939,0, > [<<"transactionByTerminal">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.paymentTerminalId) { > emit([doc.paymentTerminalId,doc.startTimestamp], null); } }">>, > {btree,<0.15650.0>, > {47574093427,{33638477,[]},942288018}, > #Fun<couch_btree.3.71804109>, > #Fun<couch_btree.4.115144917>, > #Fun<couch_view.less_json_ids.2>, > #Fun<couch_view_group.10.26766604>,snappy}, > [],[]}, > {view,2,236439939,0, > [<<"transactionBySession">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.protocolSessionId) { > emit(doc.protocolSessionId,doc.protocolTransactionId); } }">>, > {btree,<0.15650.0>, > {47574114746,{9241366,[]},131141244}, > #Fun<couch_btree.3.71804109>, > #Fun<couch_btree.4.115144917>, > #Fun<couch_view.less_json_ids.2>, > #Fun<couch_view_group.10.26766604>,snappy}, > [],[]}, > {view,3,236433956,0, > [<<"transactionByRayId">>], > <<"function(doc) { if (doc.objectType == > \"ProtocolTransaction\" && doc.cId) { emit([-(-doc.cId),doc.startTimestamp], > null); } }">>, > {btree,<0.15650.0>, > {47559121340,{2250018,[]},76590679}, > #Fun<couch_btree.3.71804109>, > #Fun<couch_btree.4.115144917>, > #Fun<couch_view.less_json_ids.2>, > #Fun<couch_view_group.10.26766604>,snappy}, > [],[]}], > {[]}, > {btree,<0.15650.0>, > {47572622835,[],1061098089}, > #Fun<couch_btree.3.71804109>, > #Fun<couch_btree.4.115144917>, > #Fun<couch_btree.5.93788370>,nil,snappy}, > 236439939,0,nil,nil}, > <0.15653.0>,nil,false, > [{{<0.15441.0>,#Ref<0.0.0.182446>},409571621}], > <0.15652.0>,false} > ** Reason for termination == > ** {os_process_error,"OS process timed out."} > [Thu, 31 May 2012 17:42:17 GMT] [error] [<0.15648.0>] {error_report,<0.31.0>, > {<0.15648.0>,crash_report, > [[{initial_call, > {couch_view_group,init,['Argument__1']}}, > {pid,<0.15648.0>}, > {registered_name,[]}, > {error_info, > {exit, > {os_process_error,"OS process timed out."}, > [{gen_server,terminate,6}, > {proc_lib,init_p_do_apply,3}]}}, > {ancestors,[<0.15647.0>]}, > {messages,[]}, > {links,[<0.15650.0>,<0.123.0>]}, > {dictionary,[]}, > {trap_exit,true}, > {status,running}, > {heap_size,2584}, > {stack_size,24}, > {reductions,18059924}], > []]}} > [Thu, 31 May 2012 17:42:17 GMT] [error] [<0.15441.0>] Uncaught server error: > {os_process_error, <<"OS process timed out.">>} > [Thu, 31 May 2012 17:42:17 GMT] [error] [<0.15650.0>] ** Generic server > <0.15650.0> terminating > ** Last message in was {'EXIT',<0.15648.0>, {os_process_error,"OS process > timed out."}} > ** When Server state == {file,{file_descriptor,prim_file,{#Port<0.2119>,19}}, > 47574426987} > ** Reason for termination == > ** {os_process_error,"OS process timed out."} > [Thu, 31 May 2012 17:42:17 GMT] [error] [<0.15650.0>] {error_report,<0.31.0>, > {<0.15650.0>,crash_report, > [[{initial_call,{couch_file,init,['Argument__1']}}, > {pid,<0.15650.0>}, > {registered_name,[]}, > {error_info, > {exit, > {os_process_error,"OS process timed out."}, > [{gen_server,terminate,6}, > {proc_lib,init_p_do_apply,3}]}}, > {ancestors,[<0.15648.0>,<0.15647.0>]}, > {messages,[{'EXIT',<0.15652.0>,shutdown}]}, > {links,[]}, > {dictionary,[]}, > {trap_exit,true}, > {status,running}, > {heap_size,2584}, > {stack_size,24}, > {reductions,27732395236}], > []]}} > > > -Sami > On Jun 5, 2012, at 12:23 PM, Dave Cottlehuber wrote: > >> On 5 June 2012 11:13, Sami Sierla <[email protected]> wrote: >>> Hi, >>> >>> We have a rather large database (about 90 million documents /200GB) running >>> on CouchDB (1.0.3) and we're now updating it to version 1.2.0 due to view >>> compaction problems (large view group compactions never finished). >>> >>> At the moment we are rebuilding (JavaScript) views with 1.2.0 but during >>> this we have stumbled upon to new problem : indexer processes suddenly just >>> disappear. Initially we got "OS Process Timeout" -errors to log but after >>> adjusting os_process_timeout to 30secs indexing still prematurely stops but >>> without any log entry. >>> >>> Any ideas what might cause this behavior? >>> >>> CouchDB is running on RHEL 5.8 and is statically linked with SpiderMonkey >>> 1.8.5 >>> >>> >>> Regards, >>> Sami Sierla / Poplatek Oy / Finland >> >> Sami, >> >> Have you anything useful in the couch.log file? Are you able to run >> the view generation in debug mode (might not be possible due to disk >> space constraints & performance impact). >> >> Also, if you query the view with ?limit=1&descending=true you'll get >> the last doc that couch successfully processed (I think). Is there >> anything special about that or the subsequent documents? If you >> process the view & those docs manually into node or js.exe directly >> [1] does that work? >> >> There's quite a few changes in 1.0.3 -> 1.2.0 including better >> detection of ill-formed docs amongst others, more info will help >> narrow this down. >> >> A+ >> Dave >> >> [1]: >> http://wiki.apache.org/couchdb/Troubleshooting#Map.2BAC8-Reduce_debugging >
