Yes, using 1.0.1 with LevelDB. I moved to it from Bitcask in the hopes of better performance.
Good to hear about your 60M key use-case. Can you share any key access performance numbers? Jim On Oct 24, 2011, at 7:23 AM, Mark Steele <[email protected]> wrote: > Just curious Kyle, you using the 1.0 series? > > I've done some informal testing on a 3 node 1.0 cluster and key listing was > working just peachy on 60 million keys using bitcask as the backend. > > Cheers, > > Mark > > On Sunday 23 October 2011 12:26:35 Aphyr wrote: >> On 10/23/2011 12:11 PM, Jim Adler wrote: >>> I will be loosening the key filter criterion after I get the basics >>> working, which I thought would be a simple equality check. 8M keys >>> isn't really a large data set, is it? I thought that keys were stored >>> in memory and key filters just operated on those memory keys and not >>> data. >>> >>> Jim >> >> That's about where we started seeing timeouts in list-keys. Around 25 >> million keys, list-keys started to take down the cluster. (6 nodes, 1024 >> partitions). You may not encounter these problems, but were I in your >> position and planning to grow... I would prepare to stop using key >> filters, bucket listing, and key listing early. >> >> Our current strategy is to store the keys in Redis, and synchronize them >> with post-commit hooks and a process that reads over bitcask. With >> ionice 3, it's fairly low-impact. https://github.com/aphyr/bitcask-ruby >> may be useful. >> >> --Kyle >> >> # Simplified code, extracted from our bitcask scanner: >> def run >> `renice 10 #{Process.pid}` >> `ionice -c 3 -p #{Process.pid}` >> >> begin >> bitcasks_dir = '/var/lib/riak/bitcask' >> dirs = Dir.entries(bitcasks_dir).select do |dir| >> dir =~ /^\d+$/ >> end.map do |dir| >> File.join(bitcasks_dir, dir) >> end >> >> dirs.each do |dir| >> scan dir >> GC.start >> end >> log.info "Completed run" >> rescue => e >> log.error "#{e}\n#{e.backtrace.join "\n"}" >> sleep 10 >> end >> end >> end >> >> def scan(dir) >> log.info "Loading #{dir}" >> b = Bitcask.new dir >> b.load >> >> log.info "Updating #{dir}" >> b.keydir.each do |key, e| >> bucket, key = BERT.decode(key).map { |x| >> Rack::Utils.unescape x >> } >> # Handle determines what to do with this particular bucket/key >> # combo; e.g. insert into redis. >> handle bucket, key, e >> end >> end >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
