## Expected Behavior Upgraded node serves document attachments as normal. ## Current Behavior After bringing a v2.2.0 node back online after upgrade, requests for document attachments will time out and event log is populated with errors. Downgrading will resolve the issue.
These errors occur with each attachment request: ``` [error] 2018-08-23T19:34:01.885601Z [email protected] <0.847.0> d2ed8a8e4d rpc couch_db:update_docs/4 function_clause [{couch_att,flush_data,[{db,1,<<"shards/6aaaaaa9-7ffffffd/gc-trails_z18_1.1528758185">>,"./data/shards/6aaaaaa9-7ffffffd/gc-trails_z18_1.1528758185.couch",{couch_bt_engine,{st,"./data/shards/6aaaaaa9-7ffffffd/gc-trails_z18_1.1528758185.couch",<0.768.0>,#Ref<0.0.31.291>,[before_header,after_header,on_file_open],{db_header,6,1554081,0,{7386710290,{676727,345228,{size_info,6238997025,6159979846}},103293828},{7386714124,1021955,95065157},{7386722545,[],3000},0,nil,7386726641,1000,<<"da56d6be35b02c52f7bf3fc765604901">>,[{'[email protected]',0}],1469805},false,{btree,<0.768.0>,{7386710290,{676727,345228,{size_info,6238997025,6159979846}},103293828},#Fun<couch_bt_engine.id_tree_split.1>,#Fun<couch_bt_engine.id_tree_join.2>,undefined,#Fun<couch_bt_engine.id_tree_reduce.2>,snappy},{btree,<0.768.0>,{7386714124,1021955,95065157},#Fun<couch_bt_engine.seq_tre e_split.1>,#Fun<couch_bt_engine.seq_tree_join.2>,undefined,#Fun<couch_bt_engine.seq_tree_reduce.2>,snappy},{btree,<0.768.0>,{7386722545,[],3000},#Fun<couch_bt_engine.local_tree_split.1>,#Fun<couch_bt_engine.local_tree_join.2>,undefined,nil,snappy},snappy}},<0.767.0>,nil,1554081,<<"1535052841840607">>,{user_ctx,null,[<<"_admin">>],undefined},[],undefined,nil,nil,nil,[{default_security_object,[]},{create_if_missing,true},{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}],undefined},{<6550.425.0>,[{7386988785,5864}]},{att,<<"tile">>,<<"image/png">>,5864,5864,<<6,194,168,120,133,24,42,241,112,11,147,174,19,248,194,60>>,1,{<6550.425.0>,[{7386988785,5864}]},identity}],[{file,"src/couch_att.erl"},{line,528}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,999}]},{couch_db,'-update_docs/4-lc$^2/1-2-',2,[{file,"src /couch_db.erl"},{line,999}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,999}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,290}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}] [error] 2018-08-23T19:34:01.885668Z [email protected] <0.846.0> 57406349f4 rpc couch_db:update_docs/4 function_clause [{couch_att,flush_data,[{db,1,<<"shards/15555555-2aaaaaa9/gc-trails_z18_1.1528758185">>,"./data/shards/15555555-2aaaaaa9/gc-trails_z18_1.1528758185.couch",{couch_bt_engine,{st,"./data/shards/15555555-2aaaaaa9/gc-trails_z18_1.1528758185.couch",<0.751.0>,#Ref<0.0.31.287>,[before_header,after_header,on_file_open],{db_header,6,1552424,0,{7407275878,{675502,344830,{size_info,6240770649,6161895863}},103735138},{7407279865,1020332,95158315},{7407284465,[],3214},0,nil,7407288561,1000,<<"d5496caef6c4e5c1b1f3113c8d2273e4">>,[{'[email protected]',0}],1468183},false,{btree,<0.751.0>,{7407275878,{675502,344830,{size_info,6240770649,6161895863}},103735138},#Fun<couch_bt_engine.id_tree_split.1>,#Fun<couch_bt_engine.id_tree_join.2>,undefined,#Fun<couch_bt_engine.id_tree_reduce.2>,snappy},{btree,<0.751.0>,{7407279865,1020332,95158315},#Fun<couch_bt_engine.seq_tre e_split.1>,#Fun<couch_bt_engine.seq_tree_join.2>,undefined,#Fun<couch_bt_engine.seq_tree_reduce.2>,snappy},{btree,<0.751.0>,{7407284465,[],3214},#Fun<couch_bt_engine.local_tree_split.1>,#Fun<couch_bt_engine.local_tree_join.2>,undefined,nil,snappy},snappy}},<0.750.0>,nil,1552424,<<"1535052841839823">>,{user_ctx,null,[<<"_admin">>],undefined},[],undefined,nil,nil,nil,[{default_security_object,[]},{create_if_missing,true},{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}],undefined},{<6550.1396.0>,[{7407505649,1039}]},{att,<<"tile">>,<<"image/png">>,1039,1039,<<126,179,48,80,135,68,57,40,80,175,38,50,198,131,24,190>>,1,{<6550.1396.0>,[{7407505649,1039}]},identity}],[{file,"src/couch_att.erl"},{line,528}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,999}]},{couch_db,'-update_docs/4-lc$^2/1-2-',2,[{file,"src /couch_db.erl"},{line,999}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,999}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,290}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}] [error] 2018-08-23T19:34:01.885944Z [email protected] <0.827.0> 3e13a3f2d9 rpc couch_db:update_docs/4 function_clause [{couch_att,flush_data,[{db,1,<<"shards/2aaaaaaa-3ffffffe/gc-trails_z17_0.1528758185">>,"./data/shards/2aaaaaaa-3ffffffe/gc-trails_z17_0.1528758185.couch",{couch_bt_engine,{st,"./data/shards/2aaaaaaa-3ffffffe/gc-trails_z17_0.1528758185.couch",<0.762.0>,#Ref<0.0.34.878>,[before_header,after_header,on_file_open],{db_header,6,560311,0,{4100113160,{265825,100701,{size_info,3355869387,3326138865}},38225741},{4100114639,366526,35123974},{4100124908,[],3967},0,nil,4100133100,1000,<<"27a814de8220280ab3f0ff7cff3096a4">>,[{'[email protected]',0}],524901},false,{btree,<0.762.0>,{4100113160,{265825,100701,{size_info,3355869387,3326138865}},38225741},#Fun<couch_bt_engine.id_tree_split.1>,#Fun<couch_bt_engine.id_tree_join.2>,undefined,#Fun<couch_bt_engine.id_tree_reduce.2>,snappy},{btree,<0.762.0>,{4100114639,366526,35123974},#Fun<couch_bt_engine.seq_tree_spli t.1>,#Fun<couch_bt_engine.seq_tree_join.2>,undefined,#Fun<couch_bt_engine.seq_tree_reduce.2>,snappy},{btree,<0.762.0>,{4100124908,[],3967},#Fun<couch_bt_engine.local_tree_split.1>,#Fun<couch_bt_engine.local_tree_join.2>,undefined,nil,snappy},snappy}},<0.761.0>,nil,560311,<<"1535052841840413">>,{user_ctx,null,[<<"_admin">>],undefined},[],undefined,nil,nil,nil,[{default_security_object,[]},{create_if_missing,true},{user_ctx,{user_ctx,null,[<<"_admin">>],undefined}}],undefined},{<6550.441.0>,[{4100354284,12062}]},{att,<<"tile">>,<<"image/png">>,12062,12062,<<172,102,80,184,139,130,156,177,143,225,5,31,174,122,99,92>>,1,{<6550.441.0>,[{4100354284,12062}]},identity}],[{file,"src/couch_att.erl"},{line,528}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,999}]},{couch_db,'-update_docs/4-lc$^2/1-2-',2,[{file,"src/c ouch_db.erl"},{line,999}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,999}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,290}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}] [error] 2018-08-23T19:34:01.886168Z [email protected] <0.356.0> d7ed63adc3 rpc couch_db:update_docs/4 function_clause [{couch_att,flush_data,[{db,1,<<"shards/55555554-6aaaaaa8/gc-trails_z18_0.1528758185">>,"./data/shards/55555554-6aaaaaa8/gc-trails_z18_0.1528758185.couch",{couch_bt_engine,{st,"./data/shards/55555554-6aaaaaa8/gc-trails_z18_0.1528758185.couch",<0.317.0>,#Ref<0.0.36.1143>,[before_header,after_header,on_file_open],{db_header,6,1551874,0,{7399201817,{675122,344475,{size_info,6231910988,6153089141}},103661925},{7399205533,1019597,95095809},{7399211249,[],2919},0,nil,7399215345,1000,<<"1a4cdc235a9d09ae96966de6453ee107">>,[{'[email protected]',0}],1467381},false,{btree,<0.317.0>,{7399201817,{675122,344475,{size_info,6231910988,6153089141}},103661925},#Fun<couch_bt_engine.id_tree_split.1>,#Fun<couch_bt_engine.id_tree_join.2>,undefined,#Fun<couch_bt_engine.id_tree_reduce.2>,snappy},{btree,<0.317.0>,{7399205533,1019597,95095809},#Fun<couch_bt_engine.seq_tr ee_split.1>,#Fun<couch_bt_engine.seq_tree_join.2>,undefined,#Fun<couch_bt_engine.seq_tree_reduce.2>,snappy},{btree,<0.317.0>,{7399211249,[],2919},#Fun<couch_bt_engine.local_tree_split.1>,#Fun<couch_bt_engine.local_tree_join.2>,undefined,nil,snappy},snappy}},<0.316.0>,nil,1551874,<<"1535052841739529">>,{user_ctx,null,[<<"_admin">>],undefined},[],undefined,nil,nil,nil,[{default_security_object,[]},{create_if_missing,true},deleted,{user_ctx,{user_ctx,null,[],undefined}},{user_ctx,{user_ctx,null,[],undefined}},attachments],undefined},{<6550.3116.0>,[{7399416049,6183}]},{att,<<"tile">>,<<"image/png">>,6183,6183,<<113,133,239,45,225,116,205,252,58,51,165,99,46,88,75,248>>,1,{<6550.3116.0>,[{7399416049,6183}]},identity}],[{file,"src/couch_att.erl"},{line,528}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,999}]}, {couch_db,'-update_docs/4-lc$^2/1-2-',2,[{file,"src/couch_db.erl"},{line,999}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,999}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,290}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}] [error] 2018-08-23T19:34:01.886419Z [email protected] <0.427.0> 42c165fbc0 rpc couch_db:update_docs/4 function_clause [{couch_att,flush_data,[{db,1,<<"shards/00000000-15555554/gc-trails_z17_1.1528758185">>,"./data/shards/00000000-15555554/gc-trails_z17_1.1528758185.couch",{couch_bt_engine,{st,"./data/shards/00000000-15555554/gc-trails_z17_1.1528758185.couch",<0.423.0>,#Ref<0.0.35.367>,[before_header,after_header,on_file_open],{db_header,6,561069,0,{3900609540,{266381,100928,{size_info,3358694892,3328899942}},38287514},{3900613463,367309,35159331},{3900621036,[],2852},0,nil,3900625132,1000,<<"2efa50ad14132f3535a0dc10eab866f1">>,[{'[email protected]',0}],525845},false,{btree,<0.423.0>,{3900609540,{266381,100928,{size_info,3358694892,3328899942}},38287514},#Fun<couch_bt_engine.id_tree_split.1>,#Fun<couch_bt_engine.id_tree_join.2>,undefined,#Fun<couch_bt_engine.id_tree_reduce.2>,snappy},{btree,<0.423.0>,{3900613463,367309,35159331},#Fun<couch_bt_engine.seq_tree_spli t.1>,#Fun<couch_bt_engine.seq_tree_join.2>,undefined,#Fun<couch_bt_engine.seq_tree_reduce.2>,snappy},{btree,<0.423.0>,{3900621036,[],2852},#Fun<couch_bt_engine.local_tree_split.1>,#Fun<couch_bt_engine.local_tree_join.2>,undefined,nil,snappy},snappy}},<0.422.0>,nil,561069,<<"1535052841755621">>,{user_ctx,null,[<<"_admin">>],undefined},[],undefined,nil,nil,nil,[{default_security_object,[]},{timeout,100},{user_ctx,{user_ctx,null,[],undefined}}],undefined},{<6550.930.0>,[{3900670188,14670}]},{att,<<"tile">>,<<"image/png">>,14670,14670,<<99,140,42,10,87,29,47,15,68,50,240,222,176,56,238,26>>,1,{<6550.930.0>,[{3900670188,14670}]},identity}],[{file,"src/couch_att.erl"},{line,528}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,999}]},{couch_db,'-update_docs/4-lc$^2/1-2-',2,[{file,"src/couch_db.erl"},{line,999}]},{ couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,999}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,290}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}] [error] 2018-08-23T19:34:01.886444Z [email protected] <0.807.0> 7ffe3f3b80 rpc couch_db:update_docs/4 function_clause [{couch_att,flush_data,[{db,1,<<"shards/2aaaaaaa-3ffffffe/gc-trails_z18_1.1528758185">>,"./data/shards/2aaaaaaa-3ffffffe/gc-trails_z18_1.1528758185.couch",{couch_bt_engine,{st,"./data/shards/2aaaaaaa-3ffffffe/gc-trails_z18_1.1528758185.couch",<0.701.0>,#Ref<0.0.27.21>,[before_header,after_header,on_file_open],{db_header,6,1549852,0,{7375990515,{675813,343692,{size_info,6228719878,6149873598}},103092634},{7375992053,1019505,94836992},{7375958257,[],3409},0,nil,7375995121,1000,<<"06f6ef6a634dfd706311d9ceb29f6539">>,[{'[email protected]',0}],1465243},false,{btree,<0.701.0>,{7375990515,{675813,343692,{size_info,6228719878,6149873598}},103092634},#Fun<couch_bt_engine.id_tree_split.1>,#Fun<couch_bt_engine.id_tree_join.2>,undefined,#Fun<couch_bt_engine.id_tree_reduce.2>,snappy},{btree,<0.701.0>,{7375992053,1019505,94836992},#Fun<couch_bt_engine.seq_tree _split.1>,#Fun<couch_bt_engine.seq_tree_join.2>,undefined,#Fun<couch_bt_engine.seq_tree_reduce.2>,snappy},{btree,<0.701.0>,{7375958257,[],3409},#Fun<couch_bt_engine.local_tree_split.1>,#Fun<couch_bt_engine.local_tree_join.2>,undefined,nil,snappy},snappy}},<0.700.0>,nil,1549852,<<"1535052841837689">>,{user_ctx,null,[<<"_admin">>],undefined},[],undefined,nil,nil,nil,[{default_security_object,[]},{create_if_missing,true},deleted,{user_ctx,{user_ctx,null,[],undefined}},{user_ctx,{user_ctx,null,[],undefined}},attachments],undefined},{<6550.3104.0>,[{7376380145,13314}]},{att,<<"tile">>,<<"image/png">>,13314,13314,<<9,197,127,77,64,114,198,218,9,235,217,195,174,31,209,145>>,1,{<6550.3104.0>,[{7376380145,13314}]},identity}],[{file,"src/couch_att.erl"},{line,528}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,doc_flush_atts,2,[{file,"src/couch_db.erl"},{line,1204}]},{couch_db,'-update_docs/4-lc$^3/1-3-',2,[{file,"src/couch_db.erl"},{line,999}] },{couch_db,'-update_docs/4-lc$^2/1-2-',2,[{file,"src/couch_db.erl"},{line,999}]},{couch_db,update_docs,4,[{file,"src/couch_db.erl"},{line,999}]},{fabric_rpc,with_db,3,[{file,"src/fabric_rpc.erl"},{line,290}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}] ``` These errors also sporadically occur: ``` [error] 2018-08-23T19:34:02.117914Z [email protected] <0.934.0> 3ef94d26ce req_err(2423162726) unknown_error : function_clause [<<"couch_att:to_binary/2 L670">>,<<"couch_att:to_json/4 L504">>,<<"couch_doc:-to_json_attachments/4-lc$^0/1-0-/4 L114">>,<<"couch_doc:to_json_attachments/4 L114">>,<<"couch_doc:doc_to_json_obj/2 L127">>,<<"chttpd_db:send_doc_efficiently/4 L916">>,<<"chttpd:handle_req_after_auth/2 L317">>,<<"chttpd:process_request/1 L300">>] [error] 2018-08-23T19:34:02.117909Z [email protected] <0.932.0> f7c107091b req_err(2423162726) unknown_error : function_clause [<<"couch_att:to_binary/2 L670">>,<<"couch_att:to_json/4 L504">>,<<"couch_doc:-to_json_attachments/4-lc$^0/1-0-/4 L114">>,<<"couch_doc:to_json_attachments/4 L114">>,<<"couch_doc:doc_to_json_obj/2 L127">>,<<"chttpd_db:send_doc_efficiently/4 L916">>,<<"chttpd:handle_req_after_auth/2 L317">>,<<"chttpd:process_request/1 L300">>] [error] 2018-08-23T19:34:02.137761Z [email protected] <0.939.0> 6640e354a0 req_err(2423162726) unknown_error : function_clause [<<"couch_att:to_binary/2 L670">>,<<"couch_att:to_json/4 L504">>,<<"couch_doc:-to_json_attachments/4-lc$^0/1-0-/4 L114">>,<<"couch_doc:to_json_attachments/4 L114">>,<<"couch_doc:doc_to_json_obj/2 L127">>,<<"chttpd_db:send_doc_efficiently/4 L916">>,<<"chttpd:handle_req_after_auth/2 L317">>,<<"chttpd:process_request/1 L300">>] [error] 2018-08-23T19:34:02.665421Z [email protected] <0.943.0> 8aa056f95f req_err(2423162726) unknown_error : function_clause [<<"couch_att:to_binary/2 L670">>,<<"couch_att:to_json/4 L504">>,<<"couch_doc:-to_json_attachments/4-lc$^0/1-0-/4 L114">>,<<"couch_doc:to_json_attachments/4 L114">>,<<"couch_doc:doc_to_json_obj/2 L127">>,<<"chttpd_db:send_doc_efficiently/4 L916">>,<<"chttpd:handle_req_after_auth/2 L317">>,<<"chttpd:process_request/1 L300">>] ``` No other errors appear in the event log. ## Steps to Reproduce (for bugs) Upgrade any node in the cluster to v2.2. ## Context <!--- How has this issue affected you? What are you trying to accomplish? --> <!--- Providing context helps us come up with a solution that is most useful in the real world --> ## Your Environment <!--- Include as many relevant details about the environment you experienced the bug in --> Six node cluster running CouchDB v2.1.1 and one node upgraded to v2.2.0 in Ubuntu 16.04. Shards evenly distributed across 4 of the 6 nodes. [ Full content available at: https://github.com/apache/couchdb/issues/1578 ] This message was relayed via gitbox.apache.org for [email protected]
