Hi Eric, How feasible is to upgrade to 1.1?
I just inserted 27547 png files to Riak using bitcask. I have a test cluster running on my laptop in a linux VM with 8 nodes and 2G ram allocated from the host to it. I am using a SSD drive and this config gives 110 inserts/s. The files are on average 150K. The CPU is pretty idle, the load is 0.3 using 4 cores. (n=3 and the ring size is 128) Cheers, I. On Thu, Feb 23, 2012 at 8:01 PM, Eric Fong <[email protected]> wrote: > Hi Russell > > I end up deleted all my old data and start from fresh db. > And use your script to put 23622 pngs (861M) into my localhost riak again. > It is much faster but also make the riak stop to response and the > riak/data folder getting larger and larger (5GB now). > Seems something wrong... > > I stopped the script but seems riak is still eating my 100% cpu and disk > space. > My server is something like 6 cpu and 48GB Ram already. *_* > > Eric > > > On Fri, Feb 24, 2012 at 10:08 AM, Eric Fong <[email protected]> wrote: > >> Hi >> >> I have a bash script put.sh: >> >> ---------------------------------------------------------------------------------------------------------- >> # upload file >> # >> # Bulk Upload: >> # put.sh local-folder bucket-name prefix-inside-bucket >> # >> file=$1 >> bucket=$2 >> prefix=$3 >> >> if [ -d "$file" ]; then >> prefixRemove=$file find $file -type f -exec $0 {} $bucket $prefix \; >> exit 0 >> fi >> >> if [ -z "$type" ]; then >> type=`file -i $file | awk '{print $2}'` >> fi >> >> # escape >> bucket=$(echo $bucket | sed 's/~/~~/g' | sed 's/\/\+/~/g' ) >> >> key=$file >> key=$(echo $key | sed 's/^\.\/\+//g') >> if [ -n "$prefixRemove" ]; then >> prefixRemove=$(echo $prefixRemove | sed 's/\(\/\)/\\\1/g') >> key=$(echo $key | eval "sed 's/$prefixRemove//g'") >> fi >> key=$(echo $prefix$key | sed 's/~/~~/g' | sed 's/\/\+/~/g') >> >> #-H "x-riak-index-date_int:`date +%s`" >> if [ -z "$test" ]; then >> curl -s -X PUT -T ${file} -H content-type:${type} >> http://magic1:8098/buckets/${bucket}/keys/${key} >> fi >> echo http://magic1:8098/buckets/${bucket}/keys/${key} >> >> ---------------------------------------------------------------------------------------------------------- >> >> >> I think it is similar to what Russell did. >> I think xargs -P is the key! >> >> I have one machine and all bucket setting is default. Then, I think the N >> is 3 >> >> >> Actually, I found another problem. Last night, I started to upload a lot >> of small pngs into riak (total size is 900MB and each of them are 10KB). >> This morning, I come back and found that riak eat up 40GB of my harddisk >> and it is 100% full. I try to restart riak few times. Still cannot get the >> whole riak work well again. I am using levelDB. Try to recover the whole >> system first. >> >> Eric >> >> >> On Thu, Feb 23, 2012 at 10:59 PM, Russell Brown <[email protected]>wrote: >> >>> >>> On 23 Feb 2012, at 14:37, Jeremiah Peschka wrote: >>> >>> Out of interest, to those of us who are rusty on their shell scripting, >>> could you share the aforementioned script? >>> >>> >>> >>> find png_test -name "*.png" -print0 | xargs -0 -P 4 -I file curl >>> -X PUT localhost:8098/riak/file --data-binary @file -H"Content-Type: >>> image/png" >>> >>> I'm a shell scripting n00b so there is probably a much better way. In >>> this case png_test is a directory I created with 1000*10k pngs. So the >>> bucket name is the directory name 'png_test'. Only works for a flat dir, it >>> was just something to check the times that Eric mentioned, please don't >>> think it is like a working script or anything. >>> >>> Cheers >>> >>> Russell >>> >>> >>> --- >>> Jeremiah Peschka >>> Managing Director, Brent Ozar PLF, LLC >>> >>> >>> On Thu, Feb 23, 2012 at 2:02 AM, Russell Brown <[email protected]>wrote: >>> >>>> Hi Eric, >>>> >>>> On 23 Feb 2012, at 08:45, Eric Fong wrote: >>>> >>>> > Hi >>>> > >>>> > Help. I want to store few thousands of small png (each of them will >>>> just be 10K) into riak. >>>> > But I find that is really slow (1 to 2 second for one png). >>>> > >>>> > Currently, I use a bash script to find and loop the directory >>>> > And curl PUT the png onto riak. >>>> > >>>> > Do we have a bulk upload or will it be faster if I use erlang or or >>>> driver? >>>> > I need to upload 1000 small png within one minute. Can it be done? >>>> >>>> I imagine it is possible. I just managed to do it in 10seconds on my >>>> laptop, single node with n=3 and r=2 using xargs and curl to put in >>>> parallel. >>>> >>>> How many nodes do you have? What is your n-val? >>>> >>>> cheers >>>> >>>> Russell >>>> >>>> > >>>> > Currently, I am on riak 1.01 and levelDB backend. >>>> > >>>> > Thanks a lot! >>>> > -- >>>> > Best Regards, >>>> > Eric Fong >>>> > _______________________________________________ >>>> > riak-users mailing list >>>> > [email protected] >>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> [email protected] >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> >> -- >> Best Regards, >> Eric Fong >> > > > > -- > Best Regards, > Eric Fong > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > -- the sun shines for all
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
