Hi All,
Also just tried to restore production data on testing cluster. As you can
see on image - http://s8.postimg.org/eu201p9d1/riak_rstore.png - it starts
at 480 op/sec and quickly drops to asymptotical 50... We have several
millions of keys, so, it will literally takes ages to restore all data.
Backing up individual bitcask/ring directories and restoring with reip
working like a charm, but require complete stop of testing cluster
operations.
With best regards,
Denis.
2013/8/8 Guillermo <[email protected]>
> Having the same problem. Still running a restore of 2.7 gb. In my case is
> 1M keys of 1kb average, and a dump file of 2.7Gb.
> Was running for at least one day already.
> The cpu in that node is 100% (seems like only one core being used).
>
> The backup was done in another cluster with smaller instances, and only
> take 15 min.
>
> Attached is riak-admin top.
> riak_control_session is always the one in the top.
>
> ---
>
> Load: cpu 8 Memory: total 256505 binary
>> 8984
>> procs 3275 processes 183808 code
>> 11393
>> runq 0 atom 613 ets
>> 9604
>> Pid Name or Initial Func Time Reds
>> Memory MsgQ Current Function
>>
>> -------------------------------------------------------------------------------------------------------------------------------
>> <6167.1386.0> riak_control_session '-' 706921
>> 972024 0 gen_server:loop/6
>> <6167.150.0> riak_core_vnode_manager '-' 47456
>> 386472 0 gen_server:loop/6
>> <6167.88.0> memsup '-' 15537
>> 88448 0 gen_server:loop/6
>> <6167.91.0> erlang:apply/2 '-' 10115
>> 2600 0 cpu_sup:measurement_server_loop/1
>> <6167.759.0> riak_kv_stat_sj_3 '-' 5542
>> 5720 0 gen_server:loop/6
>> <6167.760.0> riak_kv_stat_sj_4 '-' 4819
>> 5720 0 gen_server:loop/6
>> <6167.758.0> riak_kv_stat_sj_2 '-' 4195
>> 2704 0 gen_server:loop/6
>> <6167.734.0> riak_kv_put_fsm_sj_2 '-' 4160
>> 8736 0 gen_server:loop/6
>> <6167.80.0> riak_sysmon_filter '-' 4158
>> 13656 0 gen_server:loop/6
>> <6167.167.0> riak_core_stat_cache '-' 3293
>> 21512 0 gen_server:loop/6
>
>
>
>
>
> On Wed, Aug 7, 2013 at 10:42 PM, Justin <[email protected]> wrote:
>
>> Would you expect a restore of this size to take this long?
>>
>> My cluster is comprised of 6 nodes, bitcask, and ring size of 128.
>>
>> Total of 1.2 million keys/objects in a single bucket.
>>
>> I have some fairly large objects. A few, less than 10, are around if not
>> greater than 1GB. Less than 1000 are greater than 10MB but less than 100MB.
>>
>> Hope this helps. Thanks
>> On Aug 6, 2013 3:34 PM, "Justin" <[email protected]> wrote:
>>
>>> Hey Mark,
>>>
>>> ~ 613GB
>>>
>>> ls -lhtra /var/lib/riak/backups/all_nodes.20130725.bak
>>> -rw-r--r-- 1 riak riak 613G Jul 25 20:47
>>> /var/lib/riak/backups/all_nodes.20130725.bak
>>>
>>>
>>> On Tue, Aug 6, 2013 at 2:24 PM, Mark Phillips <[email protected]> wrote:
>>>
>>>> Hi Justin,
>>>>
>>>> For starters, how much data are you restoring?
>>>>
>>>> Mark
>>>>
>>>>
>>>> On Mon, Aug 5, 2013 at 2:46 PM, Justin <[email protected]> wrote:
>>>>
>>>>> Hello all,
>>>>>
>>>>> Is there any way to determine progress/percent complete?
>>>>>
>>>>> This has been running for 3 days now. I figured it would finish over
>>>>> the weekend but it hasn't.
>>>>>
>>>>> # riak-admin restore riak@<nodeA> riak
>>>>> /var/lib/riak/backups/all_nodes.20130725.bak
>>>>> Restoring from '/var/lib/riak/backups/all_nodes.20130725.bak' to
>>>>> cluster to which 'riak@<nodeA> belongs.
>>>>>
>>>>> I'm reluctant to kill it. It could be nearly complete OR it could be 3
>>>>> more days from finishing. The alternative is to clear my Riak cluster and
>>>>> re-import/ETL the data from scratch, which takes at least 5 days.
>>>>>
>>>>> In addition to determining progress for "restore", I'm also interested
>>>>> in determining progress for "backup".
>>>>>
>>>>> Thanks for the help.
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> [email protected]
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>
>>>>>
>>>>
>>>
>> _______________________________________________
>> riak-users mailing list
>> [email protected]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>
>
> --
> Guillermo Álvarez
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
