You're saying you want to download The Web. That's kind of ... using the optical drive tray for your coffee cup. As for hundreds of thousands of websites - there are at least 4.5 billion - and that's just the ones that have already been indexed by the insanely difficult work of search engines. That number grows constantly. You could barely even store a list of them on a 4TB drive, much less a searchable index. These things are done by huge corporate entities for a pretty good reason. Insane costs.
On Wed Feb 18 2015 at 3:14:49 AM <[email protected]> wrote: > " Worse, when a page changes you'd have to > download a whole index again." > > why? > > if you download it from a server the server could check which > pages have changed and only send those to the text file > > you definitely would not have to download the hole thing again > > "without even containing the full text to search > through - is terabytes in size." > > this may be a probelem it dosent make it impossible as you could go and > buy a > 4tb HDD > > but i guess a solution could be dividing the web pages into > 10mb or 20mb blocks > and when you want to visit a certain website you query the > server for that data it would still be anonymous > as it would be hundreds of thousands of websites > your searching though but you would not have to store a huge amount of data > > but ideally we just wait until 8tb HDD's get cheap >
