Hi Sebastian,
Thanks for the help in debugging. Turns out this issue was caused by a
recent update to our 'validate_doc_update' function. This caused an
error in the validation of 'deleted' documents, which triggered lots of
retries and ended up causing the server to hang. I'm pretty sure this
was the problem anyway, since I've fixed the validation function, the
issue has not recurred.
Conor
On 13/02/15 21:21, Sebastian Rothbucher wrote:
Hi Conor,
I'd look at the bottom of the file for two reasons
- you know the ID of the last doc from the in trace - possibly this is the
one causing the problem (and you can analyze or download+delete it)
- you'd know when in or out trace violate the one-line-in-and-one-line-out
policy of couchjs, i.e. lines are (for whichever reason) not one-liners
with valid JSON
The index is build upon requesting the view. So you have to do some change
(or at least re-submit a doc via curl) and then redo the index to cause any
map/reduce work to actually happen. I'd try to have just one indexing
process going on at a time when debugging.
Still, it's hard to tell from remote what actually this might be. A hanging
communication with couchjs might just be one reason anyway
But good luck anyways
Sebastian
On Thu, Feb 12, 2015 at 2:47 PM, Conor Mac Aoidh <[email protected]>
wrote:
Hi Sebastian,
It actually just re-occurred on one of our servers. I've applied your
logging solution and I'm getting /a lot/ of output. Any idea what Is should
be looking for in particular?
Also, when should I look? I assume the index is not built at the same time
a request is made.. maybe adding a document would trigger re-indexing?
Thanks
Conor
On 12/02/15 13:20, Conor Mac Aoidh wrote:
Hi Sebastian,
Thanks for the detailed reply! Unfortunately the problem is intermittent
and can't be triggered by the likes of stress tests, so I have to wait
until it occurs again to try your logging solution and debug further.
Thanks
Conor
On 11/02/15 21:05, Sebastian Rothbucher wrote:
Hi Conor,
it's hard to tell really. Maybe a solution is intercepting the calls to
couchjs. I just posted one possible solution here:
https://gist.github.com/sebastianrothbucher/01afe929095a55ab233e
Basically, it's using the tee command to write input and output to a file
to analyze.
I don't really have a striking idea, but maybe looking at the outputs in
your case helps.
Good luck
Sebastian
On Tue, Feb 10, 2015 at 5:00 PM, Conor Mac Aoidh <
[email protected]>
wrote:
Hi All,
Been experiencing a strange issue where a couchdb view hangs repeatedly
and does not stop hanging until the couch service is restarted. It
seems to
be happening on all databases (with the same view/design function)
simultaneously, ie. once its triggered, it happens everywhere where that
design doc is used.
Stranger still, in some of the databases that are hanging, there are
only
1 or 2 documents to index, so it does not appear to be a load issue. I
have
debug mode enabled and all I get in the logs for each request is:
/
//[Tue, 10 Feb 2015 15:33:41 GMT] [debug] [<0.1620.0>] 'GET'
/ie_conormacaoidh_gmail_com/_design/proj//ects/_view/projects/all {1,//
//1} from//
// "193.1.208.99"//
//Headers: [{'Accept',"*/*"},//
// {'Authorization',"Basic YWRtaW46UVlIOHdFcGZLQXZzUFBu"},//
// {'Host',"mysite.com:5984"},//
// {'User-Agent',"curl/7.32.0"}]//
//[Tue, 10 Feb 2015 15:33:41 GMT] [debug] [<0.1620.0>] OAuth Params: []/
And the view function:
/
//function (doc) {//
// if(doc.type === 'project' && doc.status !== 'deleted') {//
// emit(doc.created, doc);//
// }//
//}/
Anyone have any ideas what is causing this or how to debug it further?
P.S. I'm using couchdb 1.6.1
Thanks
Conor
