jvegaseg opened a new issue #1096: CouchDB 2.1.1: Sporadic Error 500 Internal Server Error when adding attachment to a document just created: fabric_worker_timeout URL: https://github.com/apache/couchdb/issues/1096 <!--- Provide a general summary of the issue in the Title above --> We have a 6-node cluster configured with q=16. We are migrating attachments from our old storage system to CouchDB. We first create the document in CouchDB database and immediately we issue a request in order to add the attachment associated to the document previously created. Everything work well but sometimes we receive a 500 Internal Server Error when trying to add an attachment. After the error, we wait 10 seconds and retry the same request and it works without problems. We would like to know the cause of these sporadic errors. Servers are very idle (CPU and disk activity). ## Expected Behavior <!--- If you're describing a bug, tell us what should happen --> <!--- If you're suggesting a change/improvement, tell us how it should work --> It should add the attachment to the document without errors. ## Current Behavior <!--- If describing a bug, tell us what happens instead of the expected behavior --> <!--- If suggesting a change/improvement, explain the difference from current behavior --> Sporadically, it gets an 500 error due to a "mfa: fabric_rpc:update_docs/3 exit:timeout". ## CouchDB logs In server who answer the request: > [error] 2018-01-07T19:12:15.517475Z [email protected] <0.14999.1143> f0ef070d10 fabric_worker_timeout update_docs,'[email protected]',<<"shards/10000000-1fffffff/ficheros.1514665102">> In other server for same request: > [error] 2018-01-07T19:21:15.517744Z [email protected] <0.25589.1086> f0ef070d10 rexi_server: from: [email protected](<13938.14999.1143>) mfa: fabric_rpc:update_docs/3 exit:timeout [{fabric_doc_attachments,'-receiver/2-fun-3-',1,[{file,"src/fabric_doc_attachments.erl"},{line,43}]},{couch_att,write_streamed_attachment,3,[{file,"src/couch_att.erl"},{line,577}]},{couch_db,with_stream,3,[{file,"src/couch_db.erl"},{line,1335}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1286}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1286}]},{couch_db,'-update_docs/4-lc$^8/1-8-',2,[{file,"src/couch_db.erl"},{line,1141}]},{couch_db,'-update_docs/4-lc$^7/1-7-',2,[{file,"src/couch_db.erl"},{line,1140}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,1140}]}] In other server for same request: > [error] 2018-01-07T19:21:15.518778Z [email protected] <0.17893.1127> f0ef070d10 rexi_server: from: [email protected](<13938.14999.1143>) mfa: fabric_rpc:update_docs/3 exit:timeout [{fabric_doc_attachments,'-receiver/2-fun-3-',1,[{file,"src/fabric_doc_attachments.erl"},{line,43}]},{couch_att,write_streamed_attachment,3,[{file,"src/couch_att.erl"},{line,577}]},{couch_db,with_stream,3,[{file,"src/couch_db.erl"},{line,1335}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1286}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1286}]},{couch_db,'-update_docs/4-lc$^8/1-8-',2,[{file,"src/couch_db.erl"},{line,1141}]},{couch_db,'-update_docs/4-lc$^7/1-7-',2,[{file,"src/couch_db.erl"},{line,1140}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,1140}]}] ## Possible Solution <!--- Not obligatory, but suggest a fix/reason for the bug, --> <!--- or ideas how to implement the addition or change --> Retrying the request works well. ## Configuration We use default settings except: local.ini ``` [cluster] q = 16 [log] level = warning ``` vm.args ``` -name [email protected] -setcookie XXXXXXXXXXXXXX -kernel inet_dist_listen_min 9100 -kernel inet_dist_listen_max 9200 -kernel error_logger silent -sasl sasl_error_logger false +K true +A 16 +Bd -noinput ``` sys.config ``` [ {lager, [ {error_logger_hwm, 1000}, {error_logger_redirect, true}, {handlers, [ {lager_console_backend, [debug, { lager_default_formatter, [ date, " ", time, " [", severity, "] ", node, " ", pid, " ", message, "\n" ] }]} ]}, {inet_dist_listen_min, 9100}, {inet_dist_listen_max, 9200} ]} ]. ``` ## Environment Software is running on virtualized Centos 7 using OpenVZ 7. Each node has same configuration. Each node has 16 cores. Some command outputs. **free -h** ``` total used free shared buff/cache available Mem: 40G 319M 28K 1,3M 39G 36G Swap: 100G 4,6M 99G ``` **top** ``` top - 22:02:37 up 9 days, 22:35, 1 user, load average: 0,41, 0,25, 0,23 Tasks: 37 total, 1 running, 36 sleeping, 0 stopped, 0 zombie %Cpu(s): 0,5 us, 0,1 sy, 0,0 ni, 99,4 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st KiB Mem : 41943040 total, 70868 free, 284816 used, 41587356 buff/cache KiB Swap: 10485760+total, 10485286+free, 4740 used. 38381588 avail Mem ``` **df -h** ``` S.ficheros Tama?o Usados Disp Uso% Montado en /dev/ploop23569p1 9,9T 72G 9,4T 1% / none 31G 0 31G 0% /sys/fs/cgroup none 31G 0 31G 0% /dev tmpfs 20G 0 20G 0% /dev/shm tmpfs 20G 144K 20G 1% /run tmpfs 4,0G 0 4,0G 0% /run/user/0 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
