Hi
I have a bash script put.sh:
----------------------------------------------------------------------------------------------------------
# upload file
#
# Bulk Upload:
# put.sh local-folder bucket-name prefix-inside-bucket
#
file=$1
bucket=$2
prefix=$3
if [ -d "$file" ]; then
prefixRemove=$file find $file -type f -exec $0 {} $bucket $prefix \;
exit 0
fi
if [ -z "$type" ]; then
type=`file -i $file | awk '{print $2}'`
fi
# escape
bucket=$(echo $bucket | sed 's/~/~~/g' | sed 's/\/\+/~/g' )
key=$file
key=$(echo $key | sed 's/^\.\/\+//g')
if [ -n "$prefixRemove" ]; then
prefixRemove=$(echo $prefixRemove | sed 's/\(\/\)/\\\1/g')
key=$(echo $key | eval "sed 's/$prefixRemove//g'")
fi
key=$(echo $prefix$key | sed 's/~/~~/g' | sed 's/\/\+/~/g')
#-H "x-riak-index-date_int:`date +%s`"
if [ -z "$test" ]; then
curl -s -X PUT -T ${file} -H content-type:${type}
http://magic1:8098/buckets/${bucket}/keys/${key}
fi
echo http://magic1:8098/buckets/${bucket}/keys/${key}
----------------------------------------------------------------------------------------------------------
I think it is similar to what Russell did.
I think xargs -P is the key!
I have one machine and all bucket setting is default. Then, I think the N
is 3
Actually, I found another problem. Last night, I started to upload a lot of
small pngs into riak (total size is 900MB and each of them are 10KB). This
morning, I come back and found that riak eat up 40GB of my harddisk and it
is 100% full. I try to restart riak few times. Still cannot get the whole
riak work well again. I am using levelDB. Try to recover the whole system
first.
Eric
On Thu, Feb 23, 2012 at 10:59 PM, Russell Brown <[email protected]> wrote:
>
> On 23 Feb 2012, at 14:37, Jeremiah Peschka wrote:
>
> Out of interest, to those of us who are rusty on their shell scripting,
> could you share the aforementioned script?
>
>
>
> find png_test -name "*.png" -print0 | xargs -0 -P 4 -I file curl -X
> PUT localhost:8098/riak/file --data-binary @file -H"Content-Type: image/png"
>
> I'm a shell scripting n00b so there is probably a much better way. In this
> case png_test is a directory I created with 1000*10k pngs. So the bucket
> name is the directory name 'png_test'. Only works for a flat dir, it was
> just something to check the times that Eric mentioned, please don't think
> it is like a working script or anything.
>
> Cheers
>
> Russell
>
>
> ---
> Jeremiah Peschka
> Managing Director, Brent Ozar PLF, LLC
>
>
> On Thu, Feb 23, 2012 at 2:02 AM, Russell Brown <[email protected]>wrote:
>
>> Hi Eric,
>>
>> On 23 Feb 2012, at 08:45, Eric Fong wrote:
>>
>> > Hi
>> >
>> > Help. I want to store few thousands of small png (each of them will
>> just be 10K) into riak.
>> > But I find that is really slow (1 to 2 second for one png).
>> >
>> > Currently, I use a bash script to find and loop the directory
>> > And curl PUT the png onto riak.
>> >
>> > Do we have a bulk upload or will it be faster if I use erlang or or
>> driver?
>> > I need to upload 1000 small png within one minute. Can it be done?
>>
>> I imagine it is possible. I just managed to do it in 10seconds on my
>> laptop, single node with n=3 and r=2 using xargs and curl to put in
>> parallel.
>>
>> How many nodes do you have? What is your n-val?
>>
>> cheers
>>
>> Russell
>>
>> >
>> > Currently, I am on riak 1.01 and levelDB backend.
>> >
>> > Thanks a lot!
>> > --
>> > Best Regards,
>> > Eric Fong
>> > _______________________________________________
>> > riak-users mailing list
>> > [email protected]
>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>>
>> _______________________________________________
>> riak-users mailing list
>> [email protected]
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
--
Best Regards,
Eric Fong
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
