Sounds like disk corruption to me.
Bryan Hunt - Client Services Engineer - Basho Technologies Limited - Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431 On 29 Jul 2014, at 13:06, Effenberg, Simon <[email protected]> wrote: > Said to say but the issue stays the same.. even after the upgrade to > 1.4.10. > > Any ideas what is happening here? > > Cheers > Simon > > On Tue, Jul 29, 2014 at 08:46:42AM +0000, Effenberg, Simon wrote: >> Already started to prepare everything for it.. :) >> >> On Tue, Jul 29, 2014 at 09:43:22AM +0100, Guido Medina wrote: >>> Hi Simon, >>> >>> There are some (maybe related) Level DB fixes in 1.4.9 and 1.4.10, I don't >>> think there isn't any harm for you to do a rolling upgrade since nothing >>> major changed, just bug fixes, here is the release notes' link for >>> reference: >>> >>> https://github.com/basho/riak/blob/1.4/RELEASE-NOTES.md >>> >>> Best regards, >>> >>> Guido. >>> >>> On 29/07/14 09:35, Effenberg, Simon wrote: >>>> Hi, >>>> >>>> we have some issues with 2i queries like that: >>>> >>>> seffenberg@kriak46-1:~$ while :; do curl -s >>>> localhost:8098/buckets/conversation/index/createdat_int/0/23182680 | ruby >>>> -rjson -e "o = JSON.parse(STDIN.read); puts o['keys'].size"; sleep 1; done >>>> >>>> 13853 >>>> 13853 >>>> 0 >>>> 557 >>>> 557 >>>> 557 >>>> 13853 >>>> 0 >>>> >>>> >>>> ... >>>> >>>> So I tried to start a repair-2i first on one vnode/partition on one node >>>> (which is quiet new in the cluster.. 2 weeks or so). >>>> >>>> The command is failing with the following log entries: >>>> >>>> seffenberg@kriak46-7:~$ sudo riak-admin repair-2i >>>> 22835963083295358096932575511191922182123945984 >>>> Will repair 2i on these partitions: >>>> 22835963083295358096932575511191922182123945984 >>>> Watch the logs for 2i repair progress reports >>>> seffenberg@kriak46-7:~$ 2014-07-29 08:20:22.729 UTC [info] >>>> <0.5929.1061>@riak_kv_2i_aae:init:139 Starting 2i repair at speed 100 for >>>> partitions [22835963083295358096932575511191922182123945984] >>>> 2014-07-29 08:20:22.729 UTC [info] >>>> <0.5930.1061>@riak_kv_2i_aae:repair_partition:257 Acquired lock on >>>> partition 22835963083295358096932575511191922182123945984 >>>> 2014-07-29 08:20:22.729 UTC [info] >>>> <0.5930.1061>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in >>>> partition 22835963083295358096932575511191922182123945984 >>>> 2014-07-29 08:20:22.740 UTC [info] >>>> <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:324 Creating temporary >>>> database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db >>>> 2014-07-29 08:20:22.751 UTC [info] >>>> <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index >>>> data for partition 22835963083295358096932575511191922182123945984 >>>> 2014-07-29 08:25:22.752 UTC [info] >>>> <0.5929.1061>@riak_kv_2i_aae:next_partition:160 Finished 2i repair: >>>> Total partitions: 1 >>>> Finished partitions: 1 >>>> Speed: 100 >>>> Total 2i items scanned: 0 >>>> Total tree objects: 0 >>>> Total objects fixed: 0 >>>> With errors: >>>> Partition: 22835963083295358096932575511191922182123945984 >>>> Error: index_scan_timeout >>>> >>>> >>>> 2014-07-29 08:25:22.752 UTC [error] <0.4711.1061> gen_server <0.4711.1061> >>>> terminated with reason: bad argument in call to >>>> eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, >>>> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], >>>> []) in eleveldb:write/3 line 155 >>>> 2014-07-29 08:25:22.753 UTC [error] <0.4711.1061> CRASH REPORT Process >>>> <0.4711.1061> with 0 neighbours exited with reason: bad argument in call >>>> to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, >>>> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], >>>> []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747 >>>> 2014-07-29 08:25:22.753 UTC [error] <0.1031.0> Supervisor >>>> {<0.1031.0>,poolboy_sup} had child riak_core_vnode_worker started with >>>> {riak_core_vnode_worker,start_link,undefined} at <0.4711.1061> exit with >>>> reason bad argument in call to >>>> eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>, >>>> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}], >>>> []) in eleveldb:write/3 line 155 in context child_terminated >>>> >>>> >>>> Anything I can do about that? What's the issue here? >>>> >>>> I'm using Riak 1.4.8 (.deb package). >>>> >>>> Cheers >>>> Simon >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> -- >> Simon Effenberg | Site Op | mobile.international GmbH >> >> Phone: + 49. 30. 8109. 7173 >> M-Phone: + 49. 151. 5266. 1558 >> Mail: [email protected] >> Web: www.mobile.de >> >> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >> >> ______________________________________________________ >> Geschäftsführer: Malte Krüger >> HRB Nr.: 18517 P, Amtsgericht Potsdam >> Sitz der Gesellschaft: Kleinmachnow >> ______________________________________________________ >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > -- > Simon Effenberg | Site Op | mobile.international GmbH > > Phone: + 49. 30. 8109. 7173 > M-Phone: + 49. 151. 5266. 1558 > Mail: [email protected] > Web: www.mobile.de > > Marktplatz 1 | 14532 Europarc Dreilinden | Germany > > ______________________________________________________ > Geschäftsführer: Malte Krüger > HRB Nr.: 18517 P, Amtsgericht Potsdam > Sitz der Gesellschaft: Kleinmachnow > ______________________________________________________ > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
