Hi Bryan,
thanks for this. I tried it but to be honest I cannot see any specific
stuff in the logs (on the specific host).
I attached the logfile from the specific node. If you think it is
also/more important to look into the logfiles on the other nodes I can
send them as well.. but a quick look into all of them (searching for
"2i" and "index") didn't show anything unusual.. the only stuff was
2014-08-07 05:44:11.298 UTC [debug]
<0.969.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=633697975561446187189878970435575840553939501056
(preflist)={610862012478150829092946394924383918371815555072,12}
and searching for errors didn't show more than you see in the attached
files:
$ for host in kriak46-{1..7} kriak47-{1..6}; do echo $host; ssh $host "grep
'^2014-08-07 05' /var/log/riak/console.log | grep -i error" ; done
kriak46-1
2014-08-07 05:38:28.596 UTC [error] <0.8949.566> ** Node
'[email protected]' not responding **
2014-08-07 05:42:36.197 UTC [error] <0.24823.566> ** Node
'[email protected]' not responding **
2014-08-07 05:43:16.213 UTC [error] <0.26434.566> ** Node
'[email protected]' not responding **
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> gen_server <0.1697.0> terminated
with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>,
<<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> CRASH REPORT Process <0.1697.0>
with 0 neighbours exited with reason: bad argument in call to
eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
2014-08-07 05:48:14.284 UTC [error] <0.1692.0> Supervisor
{<0.1692.0>,poolboy_sup} had child riak_core_vnode_worker started with
{riak_core_vnode_worker,start_link,undefined} at <0.1697.0> exit with reason
bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in context child_terminated
2014-08-i7 05:50:11.390 UTC [error] <0.20983.567> ** Node
'[email protected]' not responding **
kriak46-2
kriak46-3
kriak46-4
kriak46-5
kriak46-6
kriak46-7
kriak47-1
kriak47-2
kriak47-3
kriak47-4
kriak47-5
kriak47-6
You mentioned the partition repair stuff.. do you think I need to try
out the full repair? Is this maybe a way to fix it? Because it is quiet
hard to do this on the cluster (~15 TB of data with AAE stuff and
tombstones and maybe ~10 TB without tombstones and AAE stuff) and I
don't want to start doing this if it won't help.
Cheers
Simon
On Wed, Aug 06, 2014 at 01:08:36PM +0100, bryan hunt wrote:
> Simon,
>
> If you want to get more verbose logging information, you could perform the
> following to change the logging level, to debug, then run `repair-2i`, and
> finally switching back to the normal logging level.
>
> - `riak attach`
> - `(riak@nodename)1> SetDebug = fun() -> {node(),
> lager:set_loglevel(lager_file_backend, "/var/log/riak/console.log", debug)}
> end.`
> - `(riak@nodename)2> rp(rpc:multicall(erlang, apply, [SetDebug,[]])).`
> (don't forget the period at the end of these statements)
> - Hit CTRL+C twice to quit from the node
>
> You can then revert back to the normal `info` logging level by running the
> following command via `riak attach`:
>
> - `riak attach`
> - `(riak@nodename)1> SetInfo = fun() -> {node(),
> lager:set_loglevel(lager_file_backend, "/var/log/riak/console.log", info)}
> end.`
> - `(riak@nodename)2> rp(rpc:multicall(erlang, apply, [SetInfo,[]])).`
> (don't forget the period at the end of these statements)
> - Hit CTRL+C twice to quit from a the node
>
> Please also see the docs for info on `riak attach` monitoring of repairs.
>
> http://docs.basho.com/riak/1.4.9/ops/running/recovery/repairing-partitions/#Monitoring-Repairs
>
> Repairs can also be monitored using the `riak-admin transfers` command.
>
> http://docs.basho.com/riak/1.4.9/ops/running/recovery/repairing-partitions/#Running-a-Repair
>
> Best Regards,
>
> Bryan Hunt
>
> Bryan Hunt - Client Services Engineer - Basho Technologies Limited -
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
> On 6 Aug 2014, at 12:53, Effenberg, Simon <[email protected]> wrote:
>
> > Hi Engel,
> >
> > I tried it yesterday but it was the same:
> >
> > 2014-08-05 17:53:14.728 UTC [info]
> > <0.24306.9>@riak_kv_2i_aae:repair_partition:257 Acquired lock on partition
> > 548063113999088594326381812268606132370974703616
> > 2014-08-05 17:53:14.728 UTC [info]
> > <0.24306.9>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in
> > partition 548063113999088594326381812268606132370974703616
> > 2014-08-05 17:53:14.753 UTC [info]
> > <0.24306.9>@riak_kv_2i_aae:create_index_data_db:324 Creating temporary
> > database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
> > 2014-08-05 17:53:14.772 UTC [info]
> > <0.24306.9>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index data
> > for partition 548063113999088594326381812268606132370974703616
> > 2014-08-05 17:58:14.773 UTC [info]
> > <0.24305.9>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> > Total partitions: 1
> > Finished partitions: 1
> > Speed: 100
> > Total 2i items scanned: 0
> > Total tree objects: 0
> > Total objects fixed: 0
> > With errors:
> > Partition: 548063113999088594326381812268606132370974703616
> > Error: index_scan_timeout
> >
> > Can't we use some erlang commands to execute parts of this manually to
> > check where the timeout actually happens? Or at least who is timing out?
> >
> > Cheers
> > Simon
> >
> > On Tue, Aug 05, 2014 at 10:21:57AM -0400, Engel Sanchez wrote:
> >> Simon: The data scan for that partition seems to be taking more than 5
> >> minutes to collect a batch of 1000 items, so the 2i repair process is
> >> giving up on it before it has a chance to finish. You can reduce the
> >> likelihood of this happening by configuring the batch parameter to
> >> something small. In the riak_kv section of the configuration file, set
> >> this:
> >> {riak_kv, [
> >> {aae_2i_batch_size, 10},
> >> ...
> >> Let us know if that allows it to finish the repair. You should still
> >> look
> >> into what may be causing the slowness. A combination of slow disks or
> >> very large data sets might do it.
> >>
> >> On Fri, Aug 1, 2014 at 5:24 AM, Russell Brown <[email protected]>
> >> wrote:
> >>
> >> Hi Simon,
> >> Sorry for the delays. I'm on vacation for a couple of days. Will pick
> >> this up on Monday.
> >>
> >> Cheers
> >> Russell
> >> On 1 Aug 2014, at 09:56, Effenberg, Simon <[email protected]>
> >> wrote:
> >>
> >>> Hi Russell, @basho
> >>>
> >>> any updates on this? We still have the issues with 2i (repair is also
> >>> still not possible) and searching for the 2i indexes is reproducable
> >>> creating (for one range I tested) 3 different values.
> >>>
> >>> I would love to provide anything you need to debug that issue.
> >>>
> >>> Cheers
> >>> Simon
> >>>
> >>> On Wed, Jul 30, 2014 at 09:22:56AM +0000, Effenberg, Simon wrote:
> >>>> Great. Thanks Russell..
> >>>>
> >>>> if you need me to do something.. feel free to ask.
> >>>>
> >>>> Cheers
> >>>> Simon
> >>>>
> >>>> On Wed, Jul 30, 2014 at 10:19:56AM +0100, Russell Brown wrote:
> >>>>> Thanks Simon,
> >>>>>
> >>>>> I'm going to spend a some time on this day.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> Russell
> >>>>>
> >>>>> On 30 Jul 2014, at 10:05, Effenberg, Simon
> >> <[email protected]> wrote:
> >>>>>
> >>>>>> Hi Russel,
> >>>>>>
> >>>>>> still one machine out of 13 is on wheezy and the rest on squeeze
> >> but the
> >>>>>> software is the same and basho is providing even the erlang stuff.
> >> So
> >>>>>> their should no real difference inside the application.
> >>>>>>
> >>>>>> And the errors are almost the same (except the async_write/read
> >>>>>> difference).
> >>>>>>
> >>>>>> I paste them:
> >>>>>>
> >>>>>> ---------- node 1 -----------
> >>>>>>
> >>>>>> 2014-07-30 06:16:07.728 UTC [info]
> >> <0.14871.336>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >>>>>> Total partitions: 1
> >>>>>> Finished partitions: 1
> >>>>>> Speed: 100
> >>>>>> Total 2i items scanned: 0
> >>>>>> Total tree objects: 0
> >>>>>> Total objects fixed: 0
> >>>>>> With errors:
> >>>>>> Partition: 125597796958124469533129165311555572001681702912
> >>>>>> Error: index_scan_timeout
> >>>>>>
> >>>>>>
> >>>>>> 2014-07-30 06:16:07.728 UTC [error] <0.1525.0> gen_server
> >> <0.1525.0> terminated with reason: bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.324.211123>, <<>>,
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97
> >>>>>> ,116,105,111,110,95,115,101,99,114,...>>,...}], []) in
> >> eleveldb:write/3 line 155
> >>>>>> 2014-07-30 06:16:07.728 UTC [error] <0.1525.0> CRASH REPORT Process
> >> <0.1525.0> with 0 neighbours exited with reason: bad argument in call
> >> to
> >> eleveldb:async_write(#Ref<0.0.324.211123>, <<>>,
> >> [{put,<<131,104,2,109,0,0,0,20,99,11
> >>>>>>
> >> 1,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> >>>>>> 2014-07-30 06:16:07.728 UTC [error] <0.1517.0> Supervisor
> >> {<0.1517.0>,poolboy_sup} had child riak_core_vnode_worker started with
> >> {riak_core_vnode_worker,start_link,undefined} at <0.1525.0> exit with
> >> reason bad argument in call
> >>>>>> to eleveldb:async_write(#Ref<0.0.324.211123>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in context child_terminated
> >>>>>>
> >>>>>>
> >>>>>> ---------- node 2 -----------
> >>>>>>
> >>>>>> 2014-07-30 06:16:07.791 UTC [info]
> >> <0.8083.314>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >>>>>> Total partitions: 1
> >>>>>> Finished partitions: 1
> >>>>>> Speed: 100
> >>>>>> Total 2i items scanned: 0
> >>>>>> Total tree objects: 0
> >>>>>> Total objects fixed: 0
> >>>>>> With errors:
> >>>>>> Partition: 622279994019798508141412682679979879462877528064
> >>>>>> Error: index_scan_timeout
> >>>>>>
> >>>>>>
> >>>>>> 2014-07-30 06:16:07.791 UTC [error] <0.1884.0> gen_server
> >> <0.1884.0> terminated with reason: bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.318.96628>, <<>>,
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,
> >>>>>> 116,105,111,110,95,115,101,99,114,...>>,...}], []) in
> >> eleveldb:write/3 line 155
> >>>>>> 2014-07-30 06:16:07.791 UTC [error] <0.1884.0> CRASH REPORT Process
> >> <0.1884.0> with 0 neighbours exited with reason: bad argument in call
> >> to
> >> eleveldb:async_write(#Ref<0.0.318.96628>, <<>>,
> >> [{put,<<131,104,2,109,0,0,0,20,99,111
> >>>>>>
> >> ,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> >>>>>> 2014-07-30 06:16:07.792 UTC [error] <0.1875.0> Supervisor
> >> {<0.1875.0>,poolboy_sup} had child riak_core_vnode_worker started with
> >> {riak_core_vnode_worker,start_link,undefined} at <0.1884.0> exit with
> >> reason bad argument in call
> >>>>>> to eleveldb:async_write(#Ref<0.0.318.96628>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in context child_terminated
> >>>>>>
> >>>>>> ---------- node 3 -----------
> >>>>>>
> >>>>>> 2014-07-30 06:17:42.679 UTC [info]
> >> <0.15746.299>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >>>>>> Total partitions: 1
> >>>>>> Finished partitions: 1
> >>>>>> Speed: 100
> >>>>>> Total 2i items scanned: 0
> >>>>>> Total tree objects: 0
> >>>>>> Total objects fixed: 0
> >>>>>> With errors:
> >>>>>> Partition: 291158529312015815735890337767697007822080311296
> >>>>>> Error: index_scan_timeout
> >>>>>>
> >>>>>>
> >>>>>> 2014-07-30 06:17:42.679 UTC [error] <0.975.0> gen_server <0.975.0>
> >> terminated with reason: bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.2075.159423>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155
> >>>>>> 2014-07-30 06:17:42.679 UTC [error] <0.975.0> CRASH REPORT Process
> >> <0.975.0> with 0 neighbours exited with reason: bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.2075.159423>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> >>>>>> 2014-07-30 06:17:42.679 UTC [error] <0.969.0> Supervisor
> >> {<0.969.0>,poolboy_sup} had child riak_core_vnode_worker started with
> >> {riak_core_vnode_worker,start_link,undefined} at <0.975.0> exit with
> >> reason bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.2075.159423>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in context child_terminated
> >>>>>>
> >>>>>> ---------- node 4 -----------
> >>>>>>
> >>>>>> 2014-07-30 06:16:10.004 UTC [info]
> >> <0.28895.382>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >>>>>> Total partitions: 1
> >>>>>> Finished partitions: 1
> >>>>>> Speed: 100
> >>>>>> Total 2i items scanned: 0
> >>>>>> Total tree objects: 0
> >>>>>> Total objects fixed: 0
> >>>>>> With errors:
> >>>>>> Partition: 319703483166135013357056057156686910549735243776
> >>>>>> Error: index_scan_timeout
> >>>>>>
> >>>>>>
> >>>>>> 2014-07-30 06:16:10.004 UTC [error] <0.1580.0> gen_server
> >> <0.1580.0> terminated with reason: bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.367.155781>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155
> >>>>>> 2014-07-30 06:16:10.004 UTC [error] <0.1580.0> CRASH REPORT Process
> >> <0.1580.0> with 0 neighbours exited with reason: bad argument in call
> >> to
> >> eleveldb:async_write(#Ref<0.0.367.155781>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> >>>>>> 2014-07-30 06:16:10.005 UTC [error] <0.1570.0> Supervisor
> >> {<0.1570.0>,poolboy_sup} had child riak_core_vnode_worker started with
> >> {riak_core_vnode_worker,start_link,undefined} at <0.1580.0> exit with
> >> reason bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.367.155781>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in context child_terminated
> >>>>>>
> >>>>>> ---------- node 5 -----------
> >>>>>>
> >>>>>> 2014-07-30 06:16:09.191 UTC [info]
> >> <0.15985.355>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >>>>>> Total partitions: 1
> >>>>>> Finished partitions: 1
> >>>>>> Speed: 100
> >>>>>> Total 2i items scanned: 0
> >>>>>> Total tree objects: 0
> >>>>>> Total objects fixed: 0
> >>>>>> With errors:
> >>>>>> Partition: 833512652540280570538039006158505159647524028416
> >>>>>> Error: index_scan_timeout
> >>>>>>
> >>>>>>
> >>>>>> 2014-07-30 06:16:09.191 UTC [error] <0.1601.0> gen_server
> >> <0.1601.0> terminated with reason: bad argument in call to
> >> eleveldb:async_get(#Ref<0.0.351.26505>, <<>>,
> >>
> >> <<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,101,116,...>>,
> >> []) in eleveldb:get/3 line 143
> >>>>>> 2014-07-30 06:16:09.191 UTC [error] <0.1601.0> CRASH REPORT Process
> >> <0.1601.0> with 0 neighbours exited with reason: bad argument in call
> >> to
> >> eleveldb:async_get(#Ref<0.0.351.26505>, <<>>,
> >>
> >> <<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,101,116,...>>,
> >> []) in eleveldb:get/3 line 143 in gen_server:terminate/6 line 747
> >>>>>> 2014-07-30 06:16:09.192 UTC [error] <0.1598.0> Supervisor
> >> {<0.1598.0>,poolboy_sup} had child riak_core_vnode_worker started with
> >> {riak_core_vnode_worker,start_link,undefined} at <0.1601.0> exit with
> >> reason bad argument in call to eleveldb:async_get(#Ref<0.0.351.26505>,
> >> <<>>,
> >>
> >> <<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,101,116,...>>,
> >> []) in eleveldb:get/3 line 143 in context child_terminated
> >>>>>>
> >>>>>> ---------- node 6 -----------
> >>>>>>
> >>>>>> 2014-07-30 06:16:09.154 UTC [info]
> >> <0.32042.379>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >>>>>> Total partitions: 1
> >>>>>> Finished partitions: 1
> >>>>>> Speed: 100
> >>>>>> Total 2i items scanned: 0
> >>>>>> Total tree objects: 0
> >>>>>> Total objects fixed: 0
> >>>>>> With errors:
> >>>>>> Partition: 34253944624943037145398863266787883273185918976
> >>>>>> Error: index_scan_timeout
> >>>>>>
> >>>>>>
> >>>>>> 2014-07-30 06:16:09.154 UTC [error] <0.4086.0> gen_server
> >> <0.4086.0> terminated with reason: bad argument in call to
> >> eleveldb:async_get(#Ref<0.0.2698.198008>, <<>>,
> >>
> >> <<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,101,116,...>>,
> >> []) in eleveldb:get/3 line 143
> >>>>>> 2014-07-30 06:16:09.154 UTC [error] <0.4086.0> CRASH REPORT Process
> >> <0.4086.0> with 0 neighbours exited with reason: bad argument in call
> >> to
> >> eleveldb:async_get(#Ref<0.0.2698.198008>, <<>>,
> >>
> >> <<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,101,116,...>>,
> >> []) in eleveldb:get/3 line 143 in gen_server:terminate/6 line 747
> >>>>>> 2014-07-30 06:16:09.154 UTC [error] <0.4085.0> Supervisor
> >> {<0.4085.0>,poolboy_sup} had child riak_core_vnode_worker started with
> >> {riak_core_vnode_worker,start_link,undefined} at <0.4086.0> exit with
> >> reason bad argument in call to
> >> eleveldb:async_get(#Ref<0.0.2698.198008>,
> >> <<>>,
> >>
> >> <<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,101,116,...>>,
> >> []) in eleveldb:get/3 line 143 in context child_terminated
> >>>>>>
> >>>>>> On Wed, Jul 30, 2014 at 09:50:22AM +0100, Russell Brown wrote:
> >>>>>>> Hi Simon,
> >>>>>>> So the earlier "this is on wheezy, rest are on squeeze" thing is
> >> no longer a factor?
> >>>>>>>
> >>>>>>> Any and all 2i repair you do ends with the same error?
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> Russell
> >>>>>>>
> >>>>>>> On 30 Jul 2014, at 07:29, Effenberg, Simon
> >> <[email protected]> wrote:
> >>>>>>>
> >>>>>>>> I tried it now with one partition on 6 different machines and
> >> everywhere the same result: index_scan_timeout and the info: bad
> >> argument in call to eleveldb:async_get (2x) or async_write (4x).
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Von Samsung Mobile gesendet
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -------- Urspru:ngliche Nachricht --------
> >>>>>>>> Von: "Effenberg, Simon"
> >>>>>>>> Datum:30.07.2014 07:49 (GMT+01:00)
> >>>>>>>> An: bryan hunt
> >>>>>>>> Cc: [email protected]
> >>>>>>>> Betreff: AW: repair-2i stops with "bad argument in call to
> >> eleveldb:async_write"
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> I tried it on two different nodes with one partition each. Both
> >> multiple times before the upgrade and after the upgrade.
> >>>>>>>>
> >>>>>>>> I will try it on other machines in a minute but because I tried
> >> it already on two different nodes and one of them is 2 weeks old and
> >> stored on a HP 3par I bet that this is not a disk corruption issue..
> >>>>>>>>
> >>>>>>>> Simon
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Von Samsung Mobile gesendet
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> -------- Urspru:ngliche Nachricht --------
> >>>>>>>> Von: bryan hunt
> >>>>>>>> Datum:29.07.2014 18:21 (GMT+01:00)
> >>>>>>>> An: "Effenberg, Simon"
> >>>>>>>> Cc: [email protected]
> >>>>>>>> Betreff: Re: repair-2i stops with "bad argument in call to
> >> eleveldb:async_write"
> >>>>>>>>
> >>>>>>>> Hi Simon,
> >>>>>>>>
> >>>>>>>> Does the problem persist if you run it again?
> >>>>>>>>
> >>>>>>>> Does it happen if you run it against any other partition?
> >>>>>>>>
> >>>>>>>> Best Regards,
> >>>>>>>>
> >>>>>>>> Bryan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Bryan Hunt - Client Services Engineer - Basho Technologies
> >> Limited - Registered Office - 8 Lincoln's Inn Fields London WC2A 3BP
> >> Reg
> >> 07970431
> >>>>>>>>
> >>>>>>>> On 29 Jul 2014, at 09:35, Effenberg, Simon
> >> <[email protected]> wrote:
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> we have some issues with 2i queries like that:
> >>>>>>>>>
> >>>>>>>>> seffenberg@kriak46-1:~$ while :; do curl -s
> >> localhost:8098/buckets/conversation/index/createdat_int/0/23182680 |
> >> ruby -rjson -e "o = JSON.parse(STDIN.read); puts o['keys'].size"; sleep
> >> 1; done
> >>>>>>>>>
> >>>>>>>>> 13853
> >>>>>>>>> 13853
> >>>>>>>>> 0
> >>>>>>>>> 557
> >>>>>>>>> 557
> >>>>>>>>> 557
> >>>>>>>>> 13853
> >>>>>>>>> 0
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ...
> >>>>>>>>>
> >>>>>>>>> So I tried to start a repair-2i first on one vnode/partition on
> >> one node
> >>>>>>>>> (which is quiet new in the cluster.. 2 weeks or so).
> >>>>>>>>>
> >>>>>>>>> The command is failing with the following log entries:
> >>>>>>>>>
> >>>>>>>>> seffenberg@kriak46-7:~$ sudo riak-admin repair-2i
> >> 22835963083295358096932575511191922182123945984
> >>>>>>>>> Will repair 2i on these partitions:
> >>>>>>>>> 22835963083295358096932575511191922182123945984
> >>>>>>>>> Watch the logs for 2i repair progress reports
> >>>>>>>>> seffenberg@kriak46-7:~$ 2014-07-29 08:20:22.729 UTC [info]
> >> <0.5929.1061>@riak_kv_2i_aae:init:139 Starting 2i repair at speed 100
> >> for partitions [22835963083295358096932575511191922182123945984]
> >>>>>>>>> 2014-07-29 08:20:22.729 UTC [info]
> >> <0.5930.1061>@riak_kv_2i_aae:repair_partition:257 Acquired lock on
> >> partition 22835963083295358096932575511191922182123945984
> >>>>>>>>> 2014-07-29 08:20:22.729 UTC [info]
> >> <0.5930.1061>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in
> >> partition 22835963083295358096932575511191922182123945984
> >>>>>>>>> 2014-07-29 08:20:22.740 UTC [info]
> >> <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:324 Creating
> >> temporary
> >> database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
> >>>>>>>>> 2014-07-29 08:20:22.751 UTC [info]
> >> <0.5930.1061>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all
> >> index
> >> data for partition 22835963083295358096932575511191922182123945984
> >>>>>>>>> 2014-07-29 08:25:22.752 UTC [info]
> >> <0.5929.1061>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
> >>>>>>>>> Total partitions: 1
> >>>>>>>>> Finished partitions: 1
> >>>>>>>>> Speed: 100
> >>>>>>>>> Total 2i items scanned: 0
> >>>>>>>>> Total tree objects: 0
> >>>>>>>>> Total objects fixed: 0
> >>>>>>>>> With errors:
> >>>>>>>>> Partition: 22835963083295358096932575511191922182123945984
> >>>>>>>>> Error: index_scan_timeout
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> 2014-07-29 08:25:22.752 UTC [error] <0.4711.1061> gen_server
> >> <0.4711.1061> terminated with reason: bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155
> >>>>>>>>> 2014-07-29 08:25:22.753 UTC [error] <0.4711.1061> CRASH REPORT
> >> Process <0.4711.1061> with 0 neighbours exited with reason: bad
> >> argument
> >> in call to eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
> >>>>>>>>> 2014-07-29 08:25:22.753 UTC [error] <0.1031.0> Supervisor
> >> {<0.1031.0>,poolboy_sup} had child riak_core_vnode_worker started with
> >> {riak_core_vnode_worker,start_link,undefined} at <0.4711.1061> exit
> >> with
> >> reason bad argument in call to
> >> eleveldb:async_write(#Ref<0.0.10120.211816>, <<>>,
> >>
> >> [{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
> >> []) in eleveldb:write/3 line 155 in context child_terminated
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Anything I can do about that? What's the issue here?
> >>>>>>>>>
> >>>>>>>>> I'm using Riak 1.4.8 (.deb package).
> >>>>>>>>>
> >>>>>>>>> Cheers
> >>>>>>>>> Simon
> >>>>>>>>> _______________________________________________
> >>>>>>>>> riak-users mailing list
> >>>>>>>>> [email protected]
> >>>>>>>>>
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> riak-users mailing list
> >>>>>>>> [email protected]
> >>>>>>>>
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Simon Effenberg | Site Op | mobile.international GmbH
> >>>>>>
> >>>>>> Phone: + 49. 30. 8109. 7173
> >>>>>> M-Phone: + 49. 151. 5266. 1558
> >>>>>> Mail: [email protected]
> >>>>>> Web: www.mobile.de
> >>>>>>
> >>>>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany
> >>>>>>
> >>>>>> ______________________________________________________
> >>>>>> Gescha:ftsfu:hrer: Malte Kru:ger
> >>>>>> HRB Nr.: 18517 P, Amtsgericht Potsdam
> >>>>>> Sitz der Gesellschaft: Kleinmachnow
> >>>>>> ______________________________________________________
> >>>>>> _______________________________________________
> >>>>>> riak-users mailing list
> >>>>>> [email protected]
> >>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>>>
> >>>>
> >>>> --
> >>>> Simon Effenberg | Site Op | mobile.international GmbH
> >>>>
> >>>> Phone: + 49. 30. 8109. 7173
> >>>> M-Phone: + 49. 151. 5266. 1558
> >>>> Mail: [email protected]
> >>>> Web: www.mobile.de
> >>>>
> >>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany
> >>>>
> >>>> ______________________________________________________
> >>>> Gescha:ftsfu:hrer: Malte Kru:ger
> >>>> HRB Nr.: 18517 P, Amtsgericht Potsdam
> >>>> Sitz der Gesellschaft: Kleinmachnow
> >>>> ______________________________________________________
> >>>> _______________________________________________
> >>>> riak-users mailing list
> >>>> [email protected]
> >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>>
> >>> --
> >>> Simon Effenberg | Site Op | mobile.international GmbH
> >>>
> >>> Phone: + 49. 30. 8109. 7173
> >>> M-Phone: + 49. 151. 5266. 1558
> >>> Mail: [email protected]
> >>> Web: www.mobile.de
> >>>
> >>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany
> >>>
> >>> ______________________________________________________
> >>> Gescha:ftsfu:hrer: Malte Kru:ger
> >>> HRB Nr.: 18517 P, Amtsgericht Potsdam
> >>> Sitz der Gesellschaft: Kleinmachnow
> >>> ______________________________________________________
> >>> _______________________________________________
> >>> riak-users mailing list
> >>> [email protected]
> >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >>
> >> _______________________________________________
> >> riak-users mailing list
> >> [email protected]
> >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> > --
> > Simon Effenberg | Site Op | mobile.international GmbH
> >
> > Phone: + 49. 30. 8109. 7173
> > M-Phone: + 49. 151. 5266. 1558
> > Mail: [email protected]
> > Web: www.mobile.de
> >
> > Marktplatz 1 | 14532 Europarc Dreilinden | Germany
> >
> > ______________________________________________________
> > Geschäftsführer: Malte Krüger
> > HRB Nr.: 18517 P, Amtsgericht Potsdam
> > Sitz der Gesellschaft: Kleinmachnow
> > ______________________________________________________
> >
> > _______________________________________________
> > riak-users mailing list
> > [email protected]
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
--
Simon Effenberg | Site Op | mobile.international GmbH
Phone: + 49. 30. 8109. 7173
M-Phone: + 49. 151. 5266. 1558
Mail: [email protected]
Web: www.mobile.de
Marktplatz 1 | 14532 Europarc Dreilinden | Germany
______________________________________________________
Geschäftsführer: Malte Krüger
HRB Nr.: 18517 P, Amtsgericht Potsdam
Sitz der Gesellschaft: Kleinmachnow
______________________________________________________
2014-08-07 05:43:04.469 UTC [notice] <0.57.0>@lager_file_backend:128 Changed
loglevel of /var/log/riak/console.log to debug
2014-08-07 05:43:14.156 UTC [info] <0.27496.566>@riak_kv_2i_aae:init:139
Starting 2i repair at speed 100 for partitions
[319703483166135013357056057156686910549735243776]
2014-08-07 05:43:14.156 UTC [info]
<0.27497.566>@riak_kv_2i_aae:repair_partition:257 Acquired lock on partition
319703483166135013357056057156686910549735243776
2014-08-07 05:43:14.156 UTC [info]
<0.27497.566>@riak_kv_2i_aae:repair_partition:259 Repairing indexes in
partition 319703483166135013357056057156686910549735243776
2014-08-07 05:43:14.239 UTC [info]
<0.27497.566>@riak_kv_2i_aae:create_index_data_db:324 Creating temporary
database of 2i data in /var/lib/riak/anti_entropy/2i/tmp_db
2014-08-07 05:43:14.283 UTC [info]
<0.27497.566>@riak_kv_2i_aae:create_index_data_db:361 Grabbing all index data
for partition 319703483166135013357056057156686910549735243776
2014-08-07 05:43:14.289 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{822094670998632891489572718402909198556462055424,879184578706871286731904157180889004011771920384,{822094670998632891489572718402909198556462055424,12}}
2014-08-07 05:43:16.213 UTC [error] <0.26434.566> ** Node
'[email protected]' not responding **
** Removing (timedout) connection **
2014-08-07 05:43:29.290 UTC [debug] <0.28227.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:43:29.290 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:43:29.290 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:43:29.290 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={833512652540280570538039006158505159647524028416,12}
2014-08-07 05:43:30.633 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:43:30.633 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:43:30.633 UTC [debug]
<0.28227.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {833512652540280570538039006158505159647524028416,12}
2014-08-07 05:43:33.673 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:43:44.291 UTC [debug] <0.29028.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:43:44.291 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:43:44.291 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:43:44.291 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={839221643311104410062272150036303140193055014912,12}
2014-08-07 05:43:45.929 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:43:45.929 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:43:45.929 UTC [debug]
<0.29028.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {839221643311104410062272150036303140193055014912,12}
2014-08-07 05:43:49.312 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:43:59.292 UTC [debug] <0.29871.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:43:59.292 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:43:59.292 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:43:59.292 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={844930634081928249586505293914101120738586001408,12}
2014-08-07 05:43:59.822 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:43:59.822 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:43:59.822 UTC [debug]
<0.29871.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {844930634081928249586505293914101120738586001408,12}
2014-08-07 05:44:02.535 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:14.293 UTC [debug] <0.30743.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:14.293 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:14.293 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:44:14.293 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={850639624852752089110738437791899101284116987904,12}
2014-08-07 05:44:14.669 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
822094670998632891489572718402909198556462055424 :: normal
2014-08-07 05:44:22.126 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:44:22.126 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:44:22.126 UTC [debug]
<0.30743.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {850639624852752089110738437791899101284116987904,12}
2014-08-07 05:44:25.055 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:29.294 UTC [debug] <0.31540.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:29.294 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:29.294 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:44:29.294 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={856348615623575928634971581669697081829647974400,12}
2014-08-07 05:44:30.567 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:44:30.568 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:44:30.568 UTC [debug]
<0.31540.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {856348615623575928634971581669697081829647974400,12}
2014-08-07 05:44:33.509 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:44.295 UTC [debug] <0.32261.566>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:44.295 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:44.295 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:44:44.295 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={862057606394399768159204725547495062375178960896,12}
2014-08-07 05:44:52.849 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:44:52.849 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:44:52.849 UTC [debug]
<0.32261.566>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {862057606394399768159204725547495062375178960896,12}
2014-08-07 05:44:55.580 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:44:59.296 UTC [debug] <0.281.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:59.296 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:44:59.296 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:44:59.296 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={884893569477695126256137301058686984557302906880,3}
2014-08-07 05:45:09.895 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:45:09.895 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange between
{890602560248518965780370444936484965102833893376,'[email protected]'} and
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:45:09.895 UTC [debug]
<0.281.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {884893569477695126256137301058686984557302906880,3}
2014-08-07 05:45:11.747 UTC [info]
<0.281.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 2 keys during active
anti-entropy exchange of {884893569477695126256137301058686984557302906880,3}
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:45:11.747 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:45:14.297 UTC [debug] <0.1111.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:45:14.297 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:45:14.297 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:45:14.297 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={890602560248518965780370444936484965102833893376,3}
2014-08-07 05:45:24.617 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:45:24.617 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:45:24.617 UTC [debug]
<0.1111.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {890602560248518965780370444936484965102833893376,3}
2014-08-07 05:45:26.307 UTC [debug]
<0.1404.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=685078892498860742907977265335757665463718379520
(preflist)={673660910957213063859510977580161704372656406528,3}
2014-08-07 05:45:29.298 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,913438523331814323877303020447676887284957839360,{884893569477695126256137301058686984557302906880,12}}
2014-08-07 05:45:29.964 UTC [info]
<0.1111.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 1 keys during
active anti-entropy exchange of
{890602560248518965780370444936484965102833893376,3} between
{890602560248518965780370444936484965102833893376,'[email protected]'} and
{896311551019342805304603588814282945648364879872,'[email protected]'}
2014-08-07 05:45:29.964 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:45:44.299 UTC [debug] <0.2988.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:45:44.299 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:45:44.299 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{913438523331814323877303020447676887284957839360,'[email protected]'}
2014-08-07 05:45:44.299 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={890602560248518965780370444936484965102833893376,12}
2014-08-07 05:45:48.779 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:45:48.779 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {913438523331814323877303020447676887284957839360,'[email protected]'}
2014-08-07 05:45:48.779 UTC [debug]
<0.2988.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {890602560248518965780370444936484965102833893376,12}
2014-08-07 05:45:56.046 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:45:59.300 UTC [debug] <0.4233.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:45:59.300 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:45:59.300 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:45:59.300 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={856348615623575928634971581669697081829647974400,12}
2014-08-07 05:46:01.552 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:46:01.552 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:46:01.552 UTC [debug]
<0.4233.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {856348615623575928634971581669697081829647974400,12}
2014-08-07 05:46:06.327 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:46:14.301 UTC [debug] <0.5306.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:46:14.301 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:46:14.301 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:46:14.301 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={862057606394399768159204725547495062375178960896,12}
2014-08-07 05:46:15.236 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:46:15.236 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:46:15.236 UTC [debug]
<0.5306.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {862057606394399768159204725547495062375178960896,12}
2014-08-07 05:46:20.366 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:46:29.302 UTC [debug] <0.6563.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:46:29.302 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:46:29.302 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:46:29.302 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={873475587936047447207671013303091023466240933888,12}
2014-08-07 05:46:44.303 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,919147514102638163401536164325474867830488825856,{879184578706871286731904157180889004011771920384,12}}
2014-08-07 05:46:52.665 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:46:52.665 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:46:52.665 UTC [debug]
<0.6563.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {873475587936047447207671013303091023466240933888,12}
2014-08-07 05:46:57.385 UTC [info]
<0.6563.567>@riak_kv_exchange_fsm:key_exchange:206 Repaired 13 keys during
active anti-entropy exchange of
{873475587936047447207671013303091023466240933888,12} between
{890602560248518965780370444936484965102833893376,'[email protected]'} and
{919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:46:57.385 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:46:59.304 UTC [debug] <0.9160.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:46:59.315 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:46:59.315 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:46:59.315 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={890602560248518965780370444936484965102833893376,12}
2014-08-07 05:46:59.368 UTC [info]
<0.588.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE
throttle from 0 -> 10 msec/key, based on maximum vnode mailbox size 246 from
'[email protected]'
2014-08-07 05:47:00.465 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:47:00.465 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {919147514102638163401536164325474867830488825856,'[email protected]'}
2014-08-07 05:47:00.465 UTC [debug]
<0.9160.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {890602560248518965780370444936484965102833893376,12}
2014-08-07 05:47:05.461 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:47:14.305 UTC [debug] <0.10096.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:47:14.312 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:47:14.312 UTC [info]
<0.588.0>@riak_kv_entropy_manager:perhaps_log_throttle_change:826 Changing AAE
throttle from 10 -> 0 msec/key, based on maximum vnode mailbox size 1 from
'[email protected]'
2014-08-07 05:47:14.312 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{924856504873462002925769308203272848376019812352,'[email protected]'}
2014-08-07 05:47:14.312 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={862057606394399768159204725547495062375178960896,12}
2014-08-07 05:47:15.540 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:47:15.540 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {924856504873462002925769308203272848376019812352,'[email protected]'}
2014-08-07 05:47:15.540 UTC [debug]
<0.10096.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {862057606394399768159204725547495062375178960896,12}
2014-08-07 05:47:29.306 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,924856504873462002925769308203272848376019812352,{873475587936047447207671013303091023466240933888,12}}
2014-08-07 05:47:44.307 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:requeue_exchange:717 Requeue:
{890602560248518965780370444936484965102833893376,924856504873462002925769308203272848376019812352,{879184578706871286731904157180889004011771920384,12}}
2014-08-07 05:47:46.669 UTC [debug]
<0.588.0>@riak_kv_entropy_manager:maybe_clear_exchange:435 Untracking exchange:
890602560248518965780370444936484965102833893376 :: normal
2014-08-07 05:47:59.308 UTC [debug] <0.13725.567>@riak_kv_exchange_fsm:init:76
Starting exchange:
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:47:59.315 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:update_trees:143 Sending to
{890602560248518965780370444936484965102833893376,'[email protected]'}
2014-08-07 05:47:59.315 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:update_trees:144 Sending to
{924856504873462002925769308203272848376019812352,'[email protected]'}
2014-08-07 05:47:59.315 UTC [debug]
<0.1402.0>@riak_kv_index_hashtree:handle_call:240 Updating tree:
(vnode)=890602560248518965780370444936484965102833893376
(preflist)={884893569477695126256137301058686984557302906880,12}
2014-08-07 05:48:01.948 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:update_trees:158 Moving to key exchange
2014-08-07 05:48:01.948 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:key_exchange:171 Starting key exchange
between {890602560248518965780370444936484965102833893376,'[email protected]'}
and {924856504873462002925769308203272848376019812352,'[email protected]'}
2014-08-07 05:48:01.948 UTC [debug]
<0.13725.567>@riak_kv_exchange_fsm:key_exchange:172 Exchanging hashes for
preflist {884893569477695126256137301058686984557302906880,12}
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> gen_server <0.1697.0> terminated
with reason: bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>,
<<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155
2014-08-07 05:48:14.284 UTC [info]
<0.27496.566>@riak_kv_2i_aae:next_partition:160 Finished 2i repair:
Total partitions: 1
Finished partitions: 1
Speed: 100
Total 2i items scanned: 0
Total tree objects: 0
Total objects fixed: 0
With errors:
Partition: 319703483166135013357056057156686910549735243776
Error: index_scan_timeout
2014-08-07 05:48:14.284 UTC [error] <0.1697.0> CRASH REPORT Process <0.1697.0>
with 0 neighbours exited with reason: bad argument in call to
eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in gen_server:terminate/6 line 747
2014-08-07 05:48:14.284 UTC [error] <0.1692.0> Supervisor
{<0.1692.0>,poolboy_sup} had child riak_core_vnode_worker started with
{riak_core_vnode_worker,start_link,undefined} at <0.1697.0> exit with reason
bad argument in call to eleveldb:async_write(#Ref<0.0.567.170046>, <<>>,
[{put,<<131,104,2,109,0,0,0,20,99,111,110,118,101,114,115,97,116,105,111,110,95,115,101,99,114,...>>,...}],
[]) in eleveldb:write/3 line 155 in context child_terminated
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
