Ok, I'm starting to doubting my sanity now. I seriously hope somebody here can give me some pointer, because with the kind of behavior I'm witnessing I do not know what I could do next.
First of all, to avoid the chance that it was somewhat the server fault, or the docker fault, I installed a packaged couchdb on 2 different physical servers. On both of those, I tried replicating the hanged db, in both cases the db replicated successfully, but I was unable to rebuild the views. Once one of those view hanged, even trying to build a simple new database, with a single record and a trivial view, failed. Always the only message I see is the usual "OS process timed out". The very strange thing is that once the view hanged, even when I removed the hanged database and restarted couchdb, still I was unable to build a new view on it. After erasing everything and starting from scratch, instead of starting a replica of the db, I tried copying the records one by one. When I did this, and I tried checking the views after each single insert, the views build regularly, even quite quickly... when I tried again and instead tried to check the views only after inserting all the documents, again I met those "Os process timed out". But even this was not always the case: sometimes I build the view in a few seconds, sometimes all the db hanged up and I was unable to do anything untill I purged couchdb and restarted the machine. And this with exactly the same data, a database of 142 records, for a total size of 0.9 Mb, on different machines, used for nothing else apart these tests. I'm still believing that it's something that we're doing wrong, after all when I tried to cause these problems with "fake" data I never managed to replicate it, but really I'm becoming quite desperate... 2015-11-01 16:59 GMT+01:00 Francesco Zamboni <[email protected]>: > P.s. sorry, some technical data on the system: I'm working with a > dockerized couchdb 1.6.1, the klaemo build. > Had the same problems with a non-dockerized couchdb version 1.6.1 on a > dedicated physical machine with ubuntu 14.04, just to test for different > devices... > > > 2015-11-01 16:52 GMT+01:00 Francesco Zamboni <[email protected]>: > >> Hello to everybody: >> >> Some time ago i posted about some strange problems I had with >> replications that lead to a hanged up database. >> >> Unluckily the problem was difficult to reproduce, and none of the >> suggestion I got here managed to exactly catch the problem. >> >> After may tries, now I've concluded that replication was a red herring... >> the problems seems to be views. >> >> Sometimes they hang. >> >> The only error I always get is that unhelpful message of timeout... >> >> debug] [<0.25571.67>] OS Process Start :: #Port<0.23218> >> [debug] [<0.25571.67>] OS Process #Port<0.23218> Input :: >> ["reset",{"reduce_limit":true,"timeout":60000}] >> [error] [<0.24826.67>] OS Process Error <0.25564.67> :: {os_process_error, >> "OS process >> timed out."} >> >> I've to point that the 60000 come from an attempt to increase the timeout >> limit, but I get thousands of these lines one each few seconds, exactly the >> same as I left the 5000 default. >> >> After may attempts, it does terminate with an error report of this kind: >> >> =ERROR REPORT==== 1-Nov-2015::15:23:12 === >> Error in process <0.21125.16> with exit value: >> {{nocatch,{os_process_error,"OS process timed >> out."}},[{couch_os_process,prompt,2,[{file,"couch_os_process.erl"},{line,57}]},{couch_query_servers,map_doc_raw,2,[{file,"couch_query_servers.erl"},{line,88}]},{couch_mrview_updater... >> >> [debug] [<0.21021.16>] Minor error in HTTP request: {os_process_error, >> "OS process timed >> out."} >> [debug] [<0.21021.16>] Stacktrace: [{couch_mrview_util,get_view,4, >> [{file,"src/couch_mrview_util.erl"}, >> {line,49}]}, >> {couch_mrview,query_view,6, >> [{file,"src/couch_mrview.erl"}, >> {line,75}]}, >> {couch_httpd,etag_maybe,2, >> >> [{file,"couch_httpd.erl"},{line,610}]}, >> {couch_mrview_http,design_doc_view,5, >> [{file,"src/couch_mrview_http.erl"}, >> {line,188}]}, >> {couch_httpd_db,do_db_req,2, >> >> [{file,"couch_httpd_db.erl"},{line,234}]}, >> {couch_httpd,handle_request_int,5, >> >> [{file,"couch_httpd.erl"},{line,318}]}, >> {mochiweb_http,headers,5, >> >> [{file,"mochiweb_http.erl"},{line,94}]}, >> {proc_lib,init_p_do_apply,3, >> [{file,"proc_lib.erl"},{line,237}]}] >> [info] [<0.21021.16>] x.x.x.x - - GET /bozze/_design/libri/_view/autori >> 500 >> [error] [<0.21021.16>] httpd 500 error response: >> {"error":"os_process_error","reason":"OS process timed out."} >> >> The views that hang are really simple, they only check if a couple of >> fields are present and then emit a few fields for all the records that have >> them.. >> >> >> The record themselves are relatively tame, I tried access them one by one >> with an external script and executing externally the same code of the view, >> and all that take less than 3 seconds, json decoding included. >> >> The machine itself have plenty of spare RAM and the load while the db is >> hanged is very low, less than 1. >> >> The really frustrating thing is that there isn't anything pointing me to >> what could be wrong, if it's something I'm causing, a bug, or whatever, and >> it's making me reconsider the idea of using this product: other db systems >> too have their own quirks, but usually when they have something wrong, be >> their fault or the user, it's quite clear where the problem is... here it's >> all fog. >> >> I can understand that this is less of a problems for people that have a >> lot of experience with the product and its inner working, but for a more >> widespread use that could be a deal-breaker, nothwithstanding all the cool >> features... >> >> I'd love to be able to point to a test-case to reproduce the bug in a >> consistent manner, but all my attempts to pinpoint a cause for this >> behaviour failed: the system always hang only with real-world data, in a >> non predicable way... and as it does not point to a specific record causing >> the malfunction, I'm left incapable of finding what's causing the problem... >> >> If somebody have some suggestions, I'm otherwise at the end of what I can >> think of.. >> >> -- >> Francesco Zamboni >> >> tel: +39 0522 1590100 >> fax: +39 0522 331673 >> mob: +39 335 7548422 >> e-mail: [email protected] <[email protected]> >> web: www.mastertraining.it >> >> >> Sede Legale: via Timolini, 18 - Correggio (RE) - Italy >> Sede Operativa: via Sani, 15 - Reggio Emilia - Italy >> Sede Commerciale: via Sani, 9 - Reggio Emilia - Italy >> Le informazioni contenute in questa e-mail sono da considerarsi >> confidenziali e esclusivamente per uso personale dei destinatari sopra >> indicati. Questo messaggio può includere dati personali o sensibili. Qualora >> questo messaggio fosse da Voi ricevuto per errore vogliate cortesemente >> darcene notizia a mezzo e-mail e distruggere il messaggio ricevuto >> erroneamente. Quanto precede ai fini del rispetto del Decreto Legislativo >> 196/2003 sulla tutela dei dati personali e sensibili. >> This e-mail and any file transmitted with it is intended only for the person >> or entity to which is addressed and may contain information that is >> privileged, confidential or otherwise protected from disclosure.Copying, >> dissemination or use of this e-mail or the information herein by anyone >> other than the intended recipient is prohibited. If you have received this >> e-mail by mistake, please notify us immediately by telephone or fax. >> >> > > > -- > Francesco Zamboni > > tel: +39 0522 1590100 > fax: +39 0522 331673 > mob: +39 335 7548422 > e-mail: [email protected] <[email protected]> > web: www.mastertraining.it > > > Sede Legale: via Timolini, 18 - Correggio (RE) - Italy > Sede Operativa: via Sani, 15 - Reggio Emilia - Italy > Sede Commerciale: via Sani, 9 - Reggio Emilia - Italy > Le informazioni contenute in questa e-mail sono da considerarsi confidenziali > e esclusivamente per uso personale dei destinatari sopra indicati. Questo > messaggio può includere dati personali o sensibili. Qualora questo messaggio > fosse da Voi ricevuto per errore vogliate cortesemente darcene notizia a > mezzo e-mail e distruggere il messaggio ricevuto erroneamente. Quanto precede > ai fini del rispetto del Decreto Legislativo 196/2003 sulla tutela dei dati > personali e sensibili. > This e-mail and any file transmitted with it is intended only for the person > or entity to which is addressed and may contain information that is > privileged, confidential or otherwise protected from disclosure.Copying, > dissemination or use of this e-mail or the information herein by anyone other > than the intended recipient is prohibited. If you have received this e-mail > by mistake, please notify us immediately by telephone or fax. > > -- Francesco Zamboni tel: +39 0522 1590100 fax: +39 0522 331673 mob: +39 335 7548422 e-mail: [email protected] <[email protected]> web: www.mastertraining.it Sede Legale: via Timolini, 18 - Correggio (RE) - Italy Sede Operativa: via Sani, 15 - Reggio Emilia - Italy Sede Commerciale: via Sani, 9 - Reggio Emilia - Italy Le informazioni contenute in questa e-mail sono da considerarsi confidenziali e esclusivamente per uso personale dei destinatari sopra indicati. Questo messaggio può includere dati personali o sensibili. Qualora questo messaggio fosse da Voi ricevuto per errore vogliate cortesemente darcene notizia a mezzo e-mail e distruggere il messaggio ricevuto erroneamente. Quanto precede ai fini del rispetto del Decreto Legislativo 196/2003 sulla tutela dei dati personali e sensibili. This e-mail and any file transmitted with it is intended only for the person or entity to which is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure.Copying, dissemination or use of this e-mail or the information herein by anyone other than the intended recipient is prohibited. If you have received this e-mail by mistake, please notify us immediately by telephone or fax.
