Hi 1.1.0 version did save my life... Thanks
Eric On Fri, Feb 24, 2012 at 2:30 PM, István <[email protected]> wrote: > Hi Eric, > > How feasible is to upgrade to 1.1? > > I just inserted 27547 png files to Riak using bitcask. I have a test > cluster running on my laptop in a linux VM with 8 nodes and 2G ram > allocated from the host to it. I am using a SSD drive and this config > gives 110 inserts/s. The files are on average 150K. The CPU is pretty idle, > the load is 0.3 using 4 cores. (n=3 and the ring size is 128) > > Cheers, > I. > > > On Thu, Feb 23, 2012 at 8:01 PM, Eric Fong <[email protected]> wrote: > >> Hi Russell >> >> I end up deleted all my old data and start from fresh db. >> And use your script to put 23622 pngs (861M) into my localhost riak again. >> It is much faster but also make the riak stop to response and the >> riak/data folder getting larger and larger (5GB now). >> Seems something wrong... >> >> I stopped the script but seems riak is still eating my 100% cpu and disk >> space. >> My server is something like 6 cpu and 48GB Ram already. *_* >> >> Eric >> >> >> On Fri, Feb 24, 2012 at 10:08 AM, Eric Fong <[email protected]> wrote: >> >>> Hi >>> >>> I have a bash script put.sh: >>> >>> ---------------------------------------------------------------------------------------------------------- >>> # upload file >>> # >>> # Bulk Upload: >>> # put.sh local-folder bucket-name prefix-inside-bucket >>> # >>> file=$1 >>> bucket=$2 >>> prefix=$3 >>> >>> if [ -d "$file" ]; then >>> prefixRemove=$file find $file -type f -exec $0 {} $bucket $prefix \; >>> exit 0 >>> fi >>> >>> if [ -z "$type" ]; then >>> type=`file -i $file | awk '{print $2}'` >>> fi >>> >>> # escape >>> bucket=$(echo $bucket | sed 's/~/~~/g' | sed 's/\/\+/~/g' ) >>> >>> key=$file >>> key=$(echo $key | sed 's/^\.\/\+//g') >>> if [ -n "$prefixRemove" ]; then >>> prefixRemove=$(echo $prefixRemove | sed 's/\(\/\)/\\\1/g') >>> key=$(echo $key | eval "sed 's/$prefixRemove//g'") >>> fi >>> key=$(echo $prefix$key | sed 's/~/~~/g' | sed 's/\/\+/~/g') >>> >>> #-H "x-riak-index-date_int:`date +%s`" >>> if [ -z "$test" ]; then >>> curl -s -X PUT -T ${file} -H content-type:${type} >>> http://magic1:8098/buckets/${bucket}/keys/${key} >>> fi >>> echo http://magic1:8098/buckets/${bucket}/keys/${key} >>> >>> ---------------------------------------------------------------------------------------------------------- >>> >>> >>> I think it is similar to what Russell did. >>> I think xargs -P is the key! >>> >>> I have one machine and all bucket setting is default. Then, I think the >>> N is 3 >>> >>> >>> Actually, I found another problem. Last night, I started to upload a lot >>> of small pngs into riak (total size is 900MB and each of them are 10KB). >>> This morning, I come back and found that riak eat up 40GB of my harddisk >>> and it is 100% full. I try to restart riak few times. Still cannot get the >>> whole riak work well again. I am using levelDB. Try to recover the whole >>> system first. >>> >>> Eric >>> >>> >>> On Thu, Feb 23, 2012 at 10:59 PM, Russell Brown <[email protected]>wrote: >>> >>>> >>>> On 23 Feb 2012, at 14:37, Jeremiah Peschka wrote: >>>> >>>> Out of interest, to those of us who are rusty on their shell scripting, >>>> could you share the aforementioned script? >>>> >>>> >>>> >>>> find png_test -name "*.png" -print0 | xargs -0 -P 4 -I file curl >>>> -X PUT localhost:8098/riak/file --data-binary @file -H"Content-Type: >>>> image/png" >>>> >>>> I'm a shell scripting n00b so there is probably a much better way. In >>>> this case png_test is a directory I created with 1000*10k pngs. So the >>>> bucket name is the directory name 'png_test'. Only works for a flat dir, it >>>> was just something to check the times that Eric mentioned, please don't >>>> think it is like a working script or anything. >>>> >>>> Cheers >>>> >>>> Russell >>>> >>>> >>>> --- >>>> Jeremiah Peschka >>>> Managing Director, Brent Ozar PLF, LLC >>>> >>>> >>>> On Thu, Feb 23, 2012 at 2:02 AM, Russell Brown <[email protected]>wrote: >>>> >>>>> Hi Eric, >>>>> >>>>> On 23 Feb 2012, at 08:45, Eric Fong wrote: >>>>> >>>>> > Hi >>>>> > >>>>> > Help. I want to store few thousands of small png (each of them will >>>>> just be 10K) into riak. >>>>> > But I find that is really slow (1 to 2 second for one png). >>>>> > >>>>> > Currently, I use a bash script to find and loop the directory >>>>> > And curl PUT the png onto riak. >>>>> > >>>>> > Do we have a bulk upload or will it be faster if I use erlang or or >>>>> driver? >>>>> > I need to upload 1000 small png within one minute. Can it be done? >>>>> >>>>> I imagine it is possible. I just managed to do it in 10seconds on my >>>>> laptop, single node with n=3 and r=2 using xargs and curl to put in >>>>> parallel. >>>>> >>>>> How many nodes do you have? What is your n-val? >>>>> >>>>> cheers >>>>> >>>>> Russell >>>>> >>>>> > >>>>> > Currently, I am on riak 1.01 and levelDB backend. >>>>> > >>>>> > Thanks a lot! >>>>> > -- >>>>> > Best Regards, >>>>> > Eric Fong >>>>> > _______________________________________________ >>>>> > riak-users mailing list >>>>> > [email protected] >>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> [email protected] >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> >>> >>> -- >>> Best Regards, >>> Eric Fong >>> >> >> >> >> -- >> Best Regards, >> Eric Fong >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > > -- > the sun shines for all > > > > -- Best Regards, Eric Fong
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
