;)) On Thu, Feb 23, 2012 at 11:12 PM, Eric Fong <[email protected]> wrote:
> Hi > > 1.1.0 version did save my life... > Thanks > > Eric > > > On Fri, Feb 24, 2012 at 2:30 PM, István <[email protected]> wrote: > >> Hi Eric, >> >> How feasible is to upgrade to 1.1? >> >> I just inserted 27547 png files to Riak using bitcask. I have a test >> cluster running on my laptop in a linux VM with 8 nodes and 2G ram >> allocated from the host to it. I am using a SSD drive and this config >> gives 110 inserts/s. The files are on average 150K. The CPU is pretty idle, >> the load is 0.3 using 4 cores. (n=3 and the ring size is 128) >> >> Cheers, >> I. >> >> >> On Thu, Feb 23, 2012 at 8:01 PM, Eric Fong <[email protected]> wrote: >> >>> Hi Russell >>> >>> I end up deleted all my old data and start from fresh db. >>> And use your script to put 23622 pngs (861M) into my localhost riak >>> again. >>> It is much faster but also make the riak stop to response and the >>> riak/data folder getting larger and larger (5GB now). >>> Seems something wrong... >>> >>> I stopped the script but seems riak is still eating my 100% cpu and disk >>> space. >>> My server is something like 6 cpu and 48GB Ram already. *_* >>> >>> Eric >>> >>> >>> On Fri, Feb 24, 2012 at 10:08 AM, Eric Fong <[email protected]> wrote: >>> >>>> Hi >>>> >>>> I have a bash script put.sh: >>>> >>>> ---------------------------------------------------------------------------------------------------------- >>>> # upload file >>>> # >>>> # Bulk Upload: >>>> # put.sh local-folder bucket-name prefix-inside-bucket >>>> # >>>> file=$1 >>>> bucket=$2 >>>> prefix=$3 >>>> >>>> if [ -d "$file" ]; then >>>> prefixRemove=$file find $file -type f -exec $0 {} $bucket $prefix \; >>>> exit 0 >>>> fi >>>> >>>> if [ -z "$type" ]; then >>>> type=`file -i $file | awk '{print $2}'` >>>> fi >>>> >>>> # escape >>>> bucket=$(echo $bucket | sed 's/~/~~/g' | sed 's/\/\+/~/g' ) >>>> >>>> key=$file >>>> key=$(echo $key | sed 's/^\.\/\+//g') >>>> if [ -n "$prefixRemove" ]; then >>>> prefixRemove=$(echo $prefixRemove | sed 's/\(\/\)/\\\1/g') >>>> key=$(echo $key | eval "sed 's/$prefixRemove//g'") >>>> fi >>>> key=$(echo $prefix$key | sed 's/~/~~/g' | sed 's/\/\+/~/g') >>>> >>>> #-H "x-riak-index-date_int:`date +%s`" >>>> if [ -z "$test" ]; then >>>> curl -s -X PUT -T ${file} -H content-type:${type} >>>> http://magic1:8098/buckets/${bucket}/keys/${key} >>>> fi >>>> echo http://magic1:8098/buckets/${bucket}/keys/${key} >>>> >>>> ---------------------------------------------------------------------------------------------------------- >>>> >>>> >>>> I think it is similar to what Russell did. >>>> I think xargs -P is the key! >>>> >>>> I have one machine and all bucket setting is default. Then, I think the >>>> N is 3 >>>> >>>> >>>> Actually, I found another problem. Last night, I started to upload a >>>> lot of small pngs into riak (total size is 900MB and each of them are >>>> 10KB). This morning, I come back and found that riak eat up 40GB of my >>>> harddisk and it is 100% full. I try to restart riak few times. Still cannot >>>> get the whole riak work well again. I am using levelDB. Try to recover the >>>> whole system first. >>>> >>>> Eric >>>> >>>> >>>> On Thu, Feb 23, 2012 at 10:59 PM, Russell Brown <[email protected]>wrote: >>>> >>>>> >>>>> On 23 Feb 2012, at 14:37, Jeremiah Peschka wrote: >>>>> >>>>> Out of interest, to those of us who are rusty on their shell >>>>> scripting, could you share the aforementioned script? >>>>> >>>>> >>>>> >>>>> find png_test -name "*.png" -print0 | xargs -0 -P 4 -I file curl >>>>> -X PUT localhost:8098/riak/file --data-binary @file -H"Content-Type: >>>>> image/png" >>>>> >>>>> I'm a shell scripting n00b so there is probably a much better way. In >>>>> this case png_test is a directory I created with 1000*10k pngs. So the >>>>> bucket name is the directory name 'png_test'. Only works for a flat dir, >>>>> it >>>>> was just something to check the times that Eric mentioned, please don't >>>>> think it is like a working script or anything. >>>>> >>>>> Cheers >>>>> >>>>> Russell >>>>> >>>>> >>>>> --- >>>>> Jeremiah Peschka >>>>> Managing Director, Brent Ozar PLF, LLC >>>>> >>>>> >>>>> On Thu, Feb 23, 2012 at 2:02 AM, Russell Brown <[email protected]>wrote: >>>>> >>>>>> Hi Eric, >>>>>> >>>>>> On 23 Feb 2012, at 08:45, Eric Fong wrote: >>>>>> >>>>>> > Hi >>>>>> > >>>>>> > Help. I want to store few thousands of small png (each of them will >>>>>> just be 10K) into riak. >>>>>> > But I find that is really slow (1 to 2 second for one png). >>>>>> > >>>>>> > Currently, I use a bash script to find and loop the directory >>>>>> > And curl PUT the png onto riak. >>>>>> > >>>>>> > Do we have a bulk upload or will it be faster if I use erlang or or >>>>>> driver? >>>>>> > I need to upload 1000 small png within one minute. Can it be done? >>>>>> >>>>>> I imagine it is possible. I just managed to do it in 10seconds on my >>>>>> laptop, single node with n=3 and r=2 using xargs and curl to put in >>>>>> parallel. >>>>>> >>>>>> How many nodes do you have? What is your n-val? >>>>>> >>>>>> cheers >>>>>> >>>>>> Russell >>>>>> >>>>>> > >>>>>> > Currently, I am on riak 1.01 and levelDB backend. >>>>>> > >>>>>> > Thanks a lot! >>>>>> > -- >>>>>> > Best Regards, >>>>>> > Eric Fong >>>>>> > _______________________________________________ >>>>>> > riak-users mailing list >>>>>> > [email protected] >>>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> riak-users mailing list >>>>>> [email protected] >>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> [email protected] >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> riak-users mailing list >>>>> [email protected] >>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Eric Fong >>>> >>> >>> >>> >>> -- >>> Best Regards, >>> Eric Fong >>> >>> _______________________________________________ >>> riak-users mailing list >>> [email protected] >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> >> -- >> the sun shines for all >> >> >> >> > > > -- > Best Regards, > Eric Fong > -- the sun shines for all
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
