Hi Russell I end up deleted all my old data and start from fresh db. And use your script to put 23622 pngs (861M) into my localhost riak again. It is much faster but also make the riak stop to response and the riak/data folder getting larger and larger (5GB now). Seems something wrong...
I stopped the script but seems riak is still eating my 100% cpu and disk space. My server is something like 6 cpu and 48GB Ram already. *_* Eric On Fri, Feb 24, 2012 at 10:08 AM, Eric Fong <[email protected]> wrote: > Hi > > I have a bash script put.sh: > > ---------------------------------------------------------------------------------------------------------- > # upload file > # > # Bulk Upload: > # put.sh local-folder bucket-name prefix-inside-bucket > # > file=$1 > bucket=$2 > prefix=$3 > > if [ -d "$file" ]; then > prefixRemove=$file find $file -type f -exec $0 {} $bucket $prefix \; > exit 0 > fi > > if [ -z "$type" ]; then > type=`file -i $file | awk '{print $2}'` > fi > > # escape > bucket=$(echo $bucket | sed 's/~/~~/g' | sed 's/\/\+/~/g' ) > > key=$file > key=$(echo $key | sed 's/^\.\/\+//g') > if [ -n "$prefixRemove" ]; then > prefixRemove=$(echo $prefixRemove | sed 's/\(\/\)/\\\1/g') > key=$(echo $key | eval "sed 's/$prefixRemove//g'") > fi > key=$(echo $prefix$key | sed 's/~/~~/g' | sed 's/\/\+/~/g') > > #-H "x-riak-index-date_int:`date +%s`" > if [ -z "$test" ]; then > curl -s -X PUT -T ${file} -H content-type:${type} > http://magic1:8098/buckets/${bucket}/keys/${key} > fi > echo http://magic1:8098/buckets/${bucket}/keys/${key} > > ---------------------------------------------------------------------------------------------------------- > > > I think it is similar to what Russell did. > I think xargs -P is the key! > > I have one machine and all bucket setting is default. Then, I think the N > is 3 > > > Actually, I found another problem. Last night, I started to upload a lot > of small pngs into riak (total size is 900MB and each of them are 10KB). > This morning, I come back and found that riak eat up 40GB of my harddisk > and it is 100% full. I try to restart riak few times. Still cannot get the > whole riak work well again. I am using levelDB. Try to recover the whole > system first. > > Eric > > > On Thu, Feb 23, 2012 at 10:59 PM, Russell Brown <[email protected]>wrote: > >> >> On 23 Feb 2012, at 14:37, Jeremiah Peschka wrote: >> >> Out of interest, to those of us who are rusty on their shell scripting, >> could you share the aforementioned script? >> >> >> >> find png_test -name "*.png" -print0 | xargs -0 -P 4 -I file curl -X >> PUT localhost:8098/riak/file --data-binary @file -H"Content-Type: image/png" >> >> I'm a shell scripting n00b so there is probably a much better way. In >> this case png_test is a directory I created with 1000*10k pngs. So the >> bucket name is the directory name 'png_test'. Only works for a flat dir, it >> was just something to check the times that Eric mentioned, please don't >> think it is like a working script or anything. >> >> Cheers >> >> Russell >> >> >> --- >> Jeremiah Peschka >> Managing Director, Brent Ozar PLF, LLC >> >> >> On Thu, Feb 23, 2012 at 2:02 AM, Russell Brown <[email protected]>wrote: >> >>> Hi Eric, >>> >>> On 23 Feb 2012, at 08:45, Eric Fong wrote: >>> >>> > Hi >>> > >>> > Help. I want to store few thousands of small png (each of them will >>> just be 10K) into riak. >>> > But I find that is really slow (1 to 2 second for one png). >>> > >>> > Currently, I use a bash script to find and loop the directory >>> > And curl PUT the png onto riak. >>> > >>> > Do we have a bulk upload or will it be faster if I use erlang or or >>> driver? >>> > I need to upload 1000 small png within one minute. Can it be done? >>> >>> I imagine it is possible. I just managed to do it in 10seconds on my >>> laptop, single node with n=3 and r=2 using xargs and curl to put in >>> parallel. >>> >>> How many nodes do you have? What is your n-val? >>> >>> cheers >>> >>> Russell >>> >>> > >>> > Currently, I am on riak 1.01 and levelDB backend. >>> > >>> > Thanks a lot! >>> > -- >>> > Best Regards, >>> > Eric Fong >>> > _______________________________________________ >>> > riak-users mailing list >>> > [email protected] >>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > > -- > Best Regards, > Eric Fong > -- Best Regards, Eric Fong
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
