Solution for Slashdot Effect? By Daniel Terdiman Story location: http://www.wired.com/news/technology/0,1282,65165,00.html
02:00 AM Oct. 01, 2004 PT Anyone who's ever had their website linked on the Slashdot homepage is no doubt aware of the so-called "Slashdot effect" -- the sudden, huge rush of traffic that follows such a posting, often resulting in overloaded servers. Now, Mirrordot, a new project from the founders of network security firm Edgeos, is seeking to alleviate the Slashdot effect by automatically mirroring any website linked on the Slashdot homepage. To Jay Jacobson, the project's co-founder, anything that can reduce the frequency of Slashdot-linked sites being shut down is a good thing. "I read Slashdot all the time and always get frustrated when they post a story, the site gets flooded, and me along with thousands of other people are trying to read the content and we can't do it," Jacobson said. "It may take a day or two for things to come back to normal, and by then I've lost interest." Slashdot didn't respond to a request for comment by press time. On its FAQ page, however, the service notes that while it sees some advantage to caching some of the smaller sites it links to in order to reduce the deleterious effect the crush of traffic has on them, it has chosen not to. In part, that's because Slashdot doesn't want to hurt sites by affecting their ad revenue. In addition, Slashdot is afraid that getting permission to cache sites would take too long and would cut down on the timeliness of the stories it posts. To Jacobson, however, something had to be done. "The Slashdot effect is bad for me, it's bad for the thousands of other people who are also trying to hit that site," he said. "It's also bad for the site owner, because (the effect) drives up his bandwidth bills. Slashdot published the site for the whole purpose of letting people read it. And his content is not available." Mirrordot works by automatically copying the entire contents of a Slashdotted site to servers hosted by Deru Internet, which for now is donating the required bandwidth. Following the model of Google caching, Mirrordot puts a frame around the mirrored page in an attempt to make it clear that what it is displaying is not the original page. Further, because many sites pay for themselves through ads, Jacobson said Mirrordot maintains the original ads and their referral links to ensure that any traffic would still benefit the original sites. Then, after about two days, Jacobson said, mirrored sites would be removed from Mirrordot. However, Mirrordot can only be useful if people know to look at it for Slashdotted content. With that in mind, Jacobson said he might add comments on Slashdot postings, alerting people to the Mirrordot alternative. To Sal Cangeloso, who runs XYZ Computing, which on Wednesday was shut down for nine hours after being Slashdotted, Mirrordot offers a lot of potential. "For sites like ours, who use a high-bandwidth hosting account but are on a shared server, Mirrordot could be really great," said Cangeloso. "If we had been Mirrordotted, I am guessing (the shutdown) could have been avoided." However, Cangeloso also wonders if Mirrordot should find a way to distinguish between the smaller sites that need help and those sites that can handle the Slashdot effect. "What about larger sites who use their own dedicated servers?" Cangeloso asked. "Won't they needlessly be losing traffic when there is no Slashdot effect to solve?" Steven Grimm, a longtime Slashdot reader and poster, wondered the same thing. "There are a lot of sites that can stand up to the traffic," Grimm said, "so it would be a shame if a site is ready for that much traffic and wants it." But Grimm also said he thinks Mirrordot is a "sound" idea and that something had to be done about the Slashdot effect. "It's only going to get worse over time," he said. He also thinks, ultimately, Slashdot should bear some of the burden for solving the problem. "The real solution is for the Slashdot people to be a little more responsible and aware of the effects that they're having on the sites that they're linking to," Grimm said. "They should really be providing for small sites that they're linking to." -- You are a subscribed member of the infowarrior list. Visit www.infowarrior.org for list information or to unsubscribe. This message may be redistributed freely in its entirety. Any and all copyrights appearing in list messages are maintained by their respective owners.
