On Sat, Dec 13, 2014 at 12:20 AM, Michael Liu <[email protected]> wrote: > > I have a cluster of 2 nodes and one node was going out of memory > performing compaction. So, I did a failover, and now the remaining server > is unresponsive to my nodejs client. I have access to all my views and > documents through http rest, so I'm guessing something is wrong with the > way smart clients connect and make requests. > > In the couchdb log is full of these timeouts: > > [couchdb:error,2014-12-13T8:10:50.001,[email protected]:<0.468.0>: > couch_log:error:44]dcp client (loop, mapreduce_view: loop > _design/sync_housekeeping > (prod/main)): Obtaining mutation from server timed out after 60.0 seconds > [RequestId 123, PartId 532, StartSeq 23, EndSeq 25]. Waiting... > > In the web console I see a popup that just says Indexing all my design > docs/views, with the progress bar at 0. > > I was thinking to perform a backup/restore, but cbbbackup won't complete > either: > couchbase@union-1:~$ bin/cbbackup -v http://localhost:8091 backups/ > 2014-12-13 08:04:49,305: mt cbbackup... > 2014-12-13 08:04:49,305: mt source : http://localhost:8091 > 2014-12-13 08:04:49,305: mt sink : backups/ > 2014-12-13 08:04:49,305: mt opts : {'username': None, 'verbose': 1, > 'dry_run': False, 'extra': {'max_retry': 10.0, 'rehash': 0.0, 'data_only': > 0.0, 'uncompress': 0.0, 'nmv_retry': 1.0, 'conflict_resolve': 1.0, > 'cbb_max_mb': 100000.0, 'mcd_compatible': 1.0, 'try_xwm': 1.0, > 'backoff_cap': 10.0, 'batch_max_bytes': 400000.0, 'report_full': 2000.0, > 'seqno': 0.0, 'batch_max_size': 1000.0, 'report': 5.0, 'design_doc_only': > 0.0, 'recv_min_bytes': 4096.0}, 'single_node': False, 'vbucket_list': None > , 'threads': 4, 'mode': 'full', 'key': None, 'password': None, 'id': None, > 'bucket_source': None} > 2014-12-13 08:04:49,320: mt bucket: loop > > > > ^Cinterrupted.###### ] 83.2% (6474/estimated 7780 msgs) > > > Any help appreciated, at a loss here... >
This looks like some bug within memcached. File bug at http://www.couchbase.com/issues/secure/Dashboard.jspa. Attach cbcollectinfos and in this case it might be useful to grab backtraces of all threads in memcached. E.g. # gdb -p <pid of your memcached> > thread app all bt > quit > -- You received this message because you are subscribed to the Google Groups "Couchbase" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
