@ella: Yes I'm using 2 SSDs in parallel in the server. The SSD can do, I think, 40k IOPS. The algorithm just makes a fairly big number of random accesses for every search. Some optimization can be done there, but it will always be painful in some cases. So putting it all in RAM seems like the way to go.
I have gotten the index for the 72 GB of data down to ~100 GB, just need to fix the last bugs for the new format. I can't get AWS for lack of a credit card, but I think I don't need to. Regular dedicated servers are probably better for this application anyway. @immortal.discoveries: Are you asking this as a killer question or an actual question? I don't respond to killer questions :) @James: What good is a Google search for a bot? Google charges for every automated access. Also it's not the full search I want. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T6322565b7d29a2a0-M64291be25f5dddb81cacb392 Delivery options: https://agi.topicbox.com/groups/agi/subscription
