https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5845
--- Comment #5 from Justin Mason <[EMAIL PROTECTED]> 2008-03-15 16:57:17 PST --- (In reply to comment #4) > Woo, mapreduce! :) > > The issue with all this stuff, of course, is that it's written to deal with > processing a large dataset with processing nodes in close proximity -- ie: a > LAN w/ low latency, packet loss, high bandwidth, and probably no need for > things like SSL, etc. > > Our model is very not that. ;) not yet! This involves discussion of how to change that. For example, Amazon EC2 offers a large set of processing nodes along those lines, and I'm sure there may be other sites that would allow us to use Hadoop infrastructure "for free" as a donation. I don't think our current lack of LAN-connected nodes is a blocker. > We don't have a single dataset to run over, we have several individual (and > unlikely to be shared) datasets. Without sharing the data, we end up > immediately reducing our setup to individual processing clusters, where the > norm is to only have 1 machine (or at most a small handful of machines). As I noted, there's a public and private set of corpora. In order to adopt this, we have to be comfortable sharing a subset of our mass-checked corpora online across mass-checking nodes. I've already done this by splitting my ham corpus into "public" and "private" subsets; private stays behind on my home server, but public is already uploaded to the zone, and used in mc-fast and bb-jm mass-checks. All of my spam corpora are considered "public" in this respect. so I'd envisage this involving creating a relatively large combined corpus on the zone (or some other central point), containing everyone's "public" corpora. -- Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug.
