Thanks Kiril and Jan. @Kiril - yes thats the stock db - has ~12 million documents
@Jan - I've restored from backup (learnt the lesson some time ago and the database is replicated to two other machines). The design document contains multiple views and generating them takes a long time. thanks for your help. Sharath On Tue, Jun 9, 2020 at 5:32 PM Jan Lehnardt <[email protected]> wrote: > Hi Sharath, > > the error you are seeing originates in CouchDB’s file checksumming > feature[1]. When writing a block of data to disk, in either a > database or a view index, CouchDB also stores a checksum of the > block. When reading the block later, it checks if the stored > checksum and the newly read block still match. This is to prevent > hard drive errors sneaking into your database. > > In your case, a block in a view index doesn’t match its checksum > anymore. The reasons for this could be manifold, but all of them > will have resulted in a bit or more on your storage device having > changed values behind CouchDB’s back. > > To get back to working views, you can delete the index files and > rebuild them. > > But more importantly: > > 1. do triple check what could have caused that block to go bad. > You don’t want to have this happen to your database files. > > 2. triple check your backups are up to date and restorable. > > [1]: > https://github.com/apache/couchdb/blob/3505281559513e2922484ebf0996a8846dcc0a34/src/couch/src/couch_file.erl#L173-L188 > > Best > Jan > — > > > On 9. Jun 2020, at 02:44, Sharath <[email protected]> wrote: > > > > Hi, > > > > I'm encountering a file corruption in CouchDb 2.3.1 running on Ubuntu > > 18.04.1 LTS. > > > > The disk store is an ext4 SSD. > > > > I'm unable to access the view and couch logs shows the error below. > > > > Is there a way of knowing which view is corrupt (I have a few databases). > > > > I'm thinking of deleting all the views and recreating them - would that > > work? > > > > thanks! > > > > error 1: > > [emergency] 2020-06-09T00:33:10.122474Z [email protected] <0.1427.0> > > -------- File corruption in <0.951.0> at position 2668802729 > > [error] 2020-06-09T00:33:10.123271Z [email protected] <0.1392.0> > 9f5e486e35 > > rexi_server: from: [email protected](<0.758.0>) mfa: > fabric_rpc:map_view/5 > > throw:{file_corruption,<<"file corruption">>} > > > [{couch_mrview_util,get_view_index_state,5,[{file,"src/couch_mrview_util.erl"},{line,137}]},{couch_mrview_util,get_view,4,[{file,"src/couch_mrview_util.erl"},{line,81}]},{couch_mrview,query_view,6,[{file,"src/couch_mrview.erl"},{line,247}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,140}]}] > > [error] 2020-06-09T00:33:10.123712Z [email protected] <0.758.0> > 9f5e486e35 > > req_err(4089041121) file_corruption : file corruption > > [<<"couch_mrview_util:get_view_index_state/5 > > L137">>,<<"couch_mrview_util:get_view/4 > L81">>,<<"couch_mrview:query_view/6 > > L247">>,<<"rexi_server:init_p/3 L140">>] > > [notice] 2020-06-09T00:33:10.124065Z [email protected] <0.758.0> > 9f5e486e35 > > 192.168.0.13:5984 192.168.0.8 admin GET > > > /stock/_design/company/_view/getallamexcompanies?reduce=false&skip=0&limit=101 > > 500 ok 1111 > > [error] 2020-06-09T00:33:11.129465Z [email protected] <0.1419.0> > -------- > > gen_server <0.1419.0> terminated with reason: no match of right hand > value > > eof at couch_file:read_raw_iolist_int/3(line:627) <= > > couch_file:handle_call/3(line:449) <= > > gen_server:try_handle_call/4(line:615) <= > gen_server:handle_msg/5(line:647) > > <= proc_lib:init_p_do_apply/3(line:247) > > last msg: {pread_iolist,1190244510} > > state: > > > [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7921>,106}},false,1767501252,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/80000000-9fffffff/stock.1584663325_design/mrview/856ffe4a2101b41233877c86e8e3f8e6.view"}]}] > > extra: [] > > [error] 2020-06-09T00:33:11.132250Z [email protected] <0.1419.0> > -------- > > CRASH REPORT Process (<0.1419.0>) with 1 neighbors exited with reason: > no > > match of right hand value eof at > couch_file:read_raw_iolist_int/3(line:627) > > <= couch_file:handle_call/3(line:449) <= > > gen_server:try_handle_call/4(line:615) <= > gen_server:handle_msg/5(line:647) > > <= proc_lib:init_p_do_apply/3(line:247) at > gen_server:terminate/7(line:812) > > <= proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_file,init,['Argument__1']}, ancestors: [<0.1408.0>,<0.1407.0>], > > messages: [], links: [<0.1408.0>], dictionary: > > > [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7921>,106}},"/dat..."}},...], > > trap_exit: false, status: running, heap_size: 6772, stack_size: 27, > > reductions: 573398 > > [error] 2020-06-09T00:33:11.132607Z [email protected] <0.1432.0> > -------- > > gen_server <0.1432.0> terminated with reason: no match of right hand > value > > eof at couch_file:read_raw_iolist_int/3(line:627) <= > > couch_file:handle_call/3(line:449) <= > > gen_server:try_handle_call/4(line:615) <= > gen_server:handle_msg/5(line:647) > > <= proc_lib:init_p_do_apply/3(line:247) > > last msg: > > > {'EXIT',<0.1408.0>,{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}} > > state: {st,<0.1408.0>,couch_mrview_index,undefined} > > extra: [] > > > > > > error 2: > > [error] 2020-06-09T00:40:25.580012Z [email protected] <0.3409.0> > -------- > > gen_server <0.3409.0> terminated with reason: no match of right hand > value > > eof at couch_file:read_raw_iolist_int/3(line:627) <= > > couch_file:handle_call/3(line:449) <= > > gen_server:try_handle_call/4(line:615) <= > gen_server:handle_msg/5(line:647) > > <= proc_lib:init_p_do_apply/3(line:247) > > last msg: {pread_iolist,1223893408} > > state: > > > [{data,[{"State",{file,{file_descriptor,prim_file,{#Port<0.7858>,108}},false,1686143327,undefined,infinity}},{"InitialFilePath","/data/couchdb/.shards/00000000-1fffffff/stockanalysis.1584663342_design/mrview/98efc890b71fe5bf74611ad0868f39bf.view"}]}] > > extra: [] > > [error] 2020-06-09T00:40:25.580132Z [email protected] <0.260.0> -------- > > Compaction daemon - an error ocurred while compacting the view group > > `company` from database > > `shards/00000000-1fffffff/stockanalysis.1584663342`: > > > {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]} > > [error] 2020-06-09T00:40:25.580705Z [email protected] <0.3409.0> > -------- > > CRASH REPORT Process (<0.3409.0>) with 1 neighbors exited with reason: > no > > match of right hand value eof at > couch_file:read_raw_iolist_int/3(line:627) > > <= couch_file:handle_call/3(line:449) <= > > gen_server:try_handle_call/4(line:615) <= > gen_server:handle_msg/5(line:647) > > <= proc_lib:init_p_do_apply/3(line:247) at > gen_server:terminate/7(line:812) > > <= proc_lib:init_p_do_apply/3(line:247); initial_call: > > {couch_file,init,['Argument__1']}, ancestors: [<0.3408.0>,<0.3407.0>], > > messages: [], links: [<0.3408.0>], dictionary: > > > [{couch_file_fd,{{file_descriptor,prim_file,{#Port<0.7858>,108}},"/dat..."}},...], > > trap_exit: false, status: running, heap_size: 6772, stack_size: 27, > > reductions: 627634 > > [error] 2020-06-09T00:40:25.581361Z [email protected] emulator -------- > > Error in process <0.3415.0> on node '[email protected]' with exit value: > > > {{badmatch,{'EXIT',{{badmatch,eof},[{couch_file,read_raw_iolist_int,3,[{file,"src/couch_file.erl"},{line,627}]},{couch_file,handle_call,3,[{file,"src/couch_file.erl"},{line,449}]},{gen_server,try_handle_call,4,[{file,"gen_server.erl"},{line,615}]},{gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,647}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}},[{couch_file,pread_binary,2,[{file,"src/couch_file.erl"},{line,169}]},{couch_file,pread_term,2,[{file,"src/couch_file.erl"},{line,157}]},{couch_btree,get_node,2,[{file,"src/couch_btree.erl"},{line,434}]},{couch_btree,stream_node,7,[{file,"src/couch_btree.erl"},{line,761}]},{couch_btree,stream_kp_node,7,[{file,"src/couch_btree.erl"},{line,775}]},{couch_btree,fold,4,[{file,"src/couch_btree.erl"},{line,219}]},{couch_mrview_compactor,compact,1,[{file,"src/couch_mrview_compactor.erl"},{line,123}]},{couch_util,with_db,2,[{file,"src/couch_util.erl"},{line,556}]}]} > >
