Re: Matching infinite sets

2016-08-25 Thread Ted Mittelstaedt
On 8/22/2016 11:40 AM, Matus UHLAR - fantomas wrote: On Mon, 22 Aug 2016 09:03:38 -0700 Marc Perkel wrote: The ones that are the same are of no interest. Only where it matches one side and not the other. On 08/22/16 09:06, Dianne Skoll wrote: But... but... that's exactly like Bayes if you

Re: Matching infinite sets

2016-08-22 Thread John Hardin
On Mon, 22 Aug 2016, Matus UHLAR - fantomas wrote: > On Mon, 22 Aug 2016 09:03:38 -0700 > Marc Perkel wrote: The real magic is the feedback learning. So as it identifies ham it learns new words and phrases that then match email from other people. So it learns how normal people speak, it lea

Re: Matching infinite sets

2016-08-22 Thread Shawn Bakhtiar
On Aug 22, 2016, at 10:44 AM, Marc Perkel mailto:supp...@junkemailfilter.com>> wrote: On 08/22/16 09:06, Dianne Skoll wrote: On Mon, 22 Aug 2016 09:03:38 -0700 Marc Perkel mailto:supp...@junkemailfilter.com>> wrote: The ones that are the same are of no interest. Only where it matches one sid

Re: Matching infinite sets

2016-08-22 Thread Matus UHLAR - fantomas
On Mon, 22 Aug 2016 09:03:38 -0700 Marc Perkel wrote: The ones that are the same are of no interest. Only where it matches one side and not the other. On 08/22/16 09:06, Dianne Skoll wrote: But... but... that's exactly like Bayes if you throw out tokens whose observed probability is not 0 or

Re: Matching infinite sets

2016-08-22 Thread Dianne Skoll
On Mon, 22 Aug 2016 10:44:42 -0700 Marc Perkel wrote: > This is nothing like bayes. It's exactly like Bayes. You're stumbling across a hacked version of Bayes. You seem to lack the mathematical background to see what you're doing, thinking it's somehow fundamentally different. But it's not.

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
On 08/22/16 09:06, Dianne Skoll wrote: On Mon, 22 Aug 2016 09:03:38 -0700 Marc Perkel wrote: The ones that are the same are of no interest. Only where it matches one side and not the other. But... but... that's exactly like Bayes if you throw out tokens whose observed probability is not 0 o

Re: Matching infinite sets

2016-08-22 Thread Dianne Skoll
On Mon, 22 Aug 2016 09:06:08 -0700 Marc Perkel wrote: > Hi Dianne, what your missing are word combinations. Usually it's not > a single word but a combination of words that trigger a result. [snip] So that's Bayes with multi-word tokens, throwing out tokens whose probability is neither 0 nor 1.

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
On 08/22/16 08:58, RW wrote: On Mon, 22 Aug 2016 07:34:00 -0700 Marc Perkel wrote: On 08/22/16 07:28, Dianne Skoll wrote: The other two possibilities (no tokens in either or some tokens in both) are undecidable. Exactly! In the past you've said that when there are token in both you compare

Re: Matching infinite sets

2016-08-22 Thread Christian Grunfeld
What you are trying to do is to identify a source of messages by its entropysupposed the entropy of a ham source is distinguishable from a spam one... 2016-08-22 13:48 GMT-03:00 Antony Stone < antony.st...@spamassassin.open.source.it>: > On Monday 22 August 2016 at 18:00:35, Marc Perkel wrote

Re: Matching infinite sets

2016-08-22 Thread Antony Stone
On Monday 22 August 2016 at 18:00:35, Marc Perkel wrote: > On 08/22/16 07:37, Antony Stone wrote: > > > > So what makes "cheapest Viagra online" a token, such that "cheapest" and > > "online" are not tokens? > > They would all be tokens. Just pointing out one that would match spam > and not match

Re: Matching infinite sets

2016-08-22 Thread Dianne Skoll
On Mon, 22 Aug 2016 09:03:38 -0700 Marc Perkel wrote: > The ones that are the same are of no interest. Only where it matches > one side and not the other. But... but... that's exactly like Bayes if you throw out tokens whose observed probability is not 0 or 1. Also, in your list of tokens, they

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
On 08/22/16 07:45, Dianne Skoll wrote: On Mon, 22 Aug 2016 07:34:00 -0700 Marc Perkel wrote: So. What percentage of emails using your algorithm are actually decidable? Almost 100% if you look at a wide variety of tokens from multiple attributes. I can't believe that, or I'm missing someth

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
On 08/22/16 07:40, Antony Stone wrote: On Monday 22 August 2016 at 16:34:00, Marc Perkel wrote: On 08/22/16 07:28, Dianne Skoll wrote: What percentage of emails using your algorithm are actually decidable? Almost 100% if you look at a wide variety of tokens from multiple attributes. Subjec

Re: Matching infinite sets

2016-08-22 Thread Shawn Bakhtiar
> On Aug 22, 2016, at 8:09 AM, John Hardin wrote: > > On Mon, 22 Aug 2016, Antony Stone wrote: > >> On Monday 22 August 2016 at 16:45:09, Dianne Skoll wrote: >> >>> On Mon, 22 Aug 2016 07:34:00 -0700 Marc Perkel wrote: > So. What percentage of emails using your algorithm are actually

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
On 08/22/16 07:37, Antony Stone wrote: On Monday 22 August 2016 at 16:34:09, Marc Perkel wrote: OK - Trying to make the really simple. Just talking about concept now. Let's say I get an email where the subject is "I have aednocarsonoma of the lung". Right off you know it's ham because spamm

Re: Matching infinite sets

2016-08-22 Thread RW
On Mon, 22 Aug 2016 07:34:00 -0700 Marc Perkel wrote: > On 08/22/16 07:28, Dianne Skoll wrote: > > The other two possibilities (no tokens in either or some tokens in > > both) are undecidable. > > Exactly! In the past you've said that when there are token in both you compare the counts. On

Re: Matching infinite sets

2016-08-22 Thread John Hardin
On Mon, 22 Aug 2016, Antony Stone wrote: On Monday 22 August 2016 at 16:45:09, Dianne Skoll wrote: On Mon, 22 Aug 2016 07:34:00 -0700 Marc Perkel wrote: So. What percentage of emails using your algorithm are actually decidable? Almost 100% if you look at a wide variety of tokens from multi

Re: Matching infinite sets

2016-08-22 Thread Antony Stone
On Monday 22 August 2016 at 16:45:09, Dianne Skoll wrote: > On Mon, 22 Aug 2016 07:34:00 -0700 Marc Perkel wrote: > > > So. What percentage of emails using your algorithm are actually > > > decidable? > > > > Almost 100% if you look at a wide variety of tokens from multiple > > attributes. > >

Re: Matching infinite sets

2016-08-22 Thread Dianne Skoll
On Mon, 22 Aug 2016 07:34:00 -0700 Marc Perkel wrote: > > So. What percentage of emails using your algorithm are actually > > decidable? > Almost 100% if you look at a wide variety of tokens from multiple > attributes. I can't believe that, or I'm missing something. Almost every spam I see c

Re: Matching infinite sets

2016-08-22 Thread Antony Stone
On Monday 22 August 2016 at 16:34:00, Marc Perkel wrote: > On 08/22/16 07:28, Dianne Skoll wrote: > > > What percentage of emails using your algorithm are actually > > decidable? > > Almost 100% if you look at a wide variety of tokens from multiple > attributes. Subject, body, content flags, hea

Re: Matching infinite sets

2016-08-22 Thread Antony Stone
On Monday 22 August 2016 at 16:34:09, Marc Perkel wrote: > OK - Trying to make the really simple. Just talking about concept now. > > Let's say I get an email where the subject is "I have aednocarsonoma of > the lung". > > Right off you know it's ham because spammers never use the word > "aednoc

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
OK - Trying to make the really simple. Just talking about concept now. Let's say I get an email where the subject is "I have aednocarsonoma of the lung". Right off you know it's ham because spammers never use the word "aednocarsonoma" and normal people do. Spammer also never use: "of the lu

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
On 08/22/16 07:28, Dianne Skoll wrote: On Mon, 22 Aug 2016 07:16:41 -0700 Marc Perkel wrote: Anthony, Yes - I don't store Set B. I store Set A. B is defined by what's NOT in A. So I test A and if it's not matched it's set B. Set B is just a negative match on A. Let me ask you a question. A

Re: Matching infinite sets

2016-08-22 Thread Dianne Skoll
On Mon, 22 Aug 2016 07:16:41 -0700 Marc Perkel wrote: > Anthony, Yes - I don't store Set B. I store Set A. B is defined by > what's NOT in A. So I test A and if it's not matched it's set B. Set > B is just a negative match on A. Let me ask you a question. As far as I understand your algorithm,

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
On 08/22/16 06:55, Antony Stone wrote: On Monday 22 August 2016 at 15:46:41, Dianne Skoll wrote: On Mon, 22 Aug 2016 06:04:49 -0700 Marc Perkel wrote: Set A - a finite set - has some members, Set B - an infinite set - is everything that is NOT in Set A Set B is a very special case of an

Re: Matching infinite sets

2016-08-22 Thread Antony Stone
On Monday 22 August 2016 at 15:46:41, Dianne Skoll wrote: > On Mon, 22 Aug 2016 06:04:49 -0700 > > Marc Perkel wrote: > > Set A - a finite set - has some members, > > Set B - an infinite set - is everything that is NOT in Set A > > Set B is a very special case of an infinite set. We're talkin

Re: Matching infinite sets

2016-08-22 Thread Dianne Skoll
On Mon, 22 Aug 2016 06:04:49 -0700 Marc Perkel wrote: > Set A - a finite set - has some members, > Set B - and infinite set - is everything that is NOT in Set A Set B is a very special case of an infinite set. We're talking about infinite sets in general. Also, you have to realize that althou

Re: Matching infinite sets

2016-08-22 Thread Dianne Skoll
On Mon, 22 Aug 2016 08:54:48 -0400 Michael Orlitzky wrote: > The empty set contains itself. No, it doesn't. By definition. Regards, Dianne.

Re: Matching infinite sets

2016-08-22 Thread Antony Stone
On Monday 22 August 2016 at 15:04:49, Marc Perkel wrote: > I'm confused by the confusion here. > > Set A - a finite set - has some members, > Set B - and infinite set - is everything that is NOT in Set A > > So you match a test item to Set A and if it matches it's a member of A. > If it doesn't

Re: Matching infinite sets

2016-08-22 Thread RW
On Mon, 22 Aug 2016 09:55:10 +1200 Sidney Markowitz wrote: > I'm one of those people he mentions who understands > how Bayesian spam filtering works who has yet to wrap my head around > what he is presenting - For now I'm staying agnostic about it until I > do understand it better). What it amo

Re: Matching infinite sets

2016-08-22 Thread Michael Orlitzky
On 08/22/2016 09:02 AM, Joe Quinn wrote: > On 8/22/2016 8:54 AM, Michael Orlitzky wrote: >> On 08/21/2016 03:22 PM, Damian wrote: >>> There is no such set B, as it would contain itself. >> The empty set contains itself. > That's an easy mistake to make. The empty set is {}, the set that > contains

Re: Matching infinite sets

2016-08-22 Thread Marc Perkel
I'm confused by the confusion here. Set A - a finite set - has some members, Set B - and infinite set - is everything that is NOT in Set A So you match a test item to Set A and if it matches it's a member of A. If it doesn't match Set A it's a member of B. How is this not really simple?

Re: Matching infinite sets

2016-08-22 Thread Joe Quinn
On 8/22/2016 8:54 AM, Michael Orlitzky wrote: On 08/21/2016 03:22 PM, Damian wrote: There is no such set B, as it would contain itself. The empty set contains itself. That's an easy mistake to make. The empty set is {}, the set that contains only the empty set is {{}}. Sets are discrete elemen

Re: Matching infinite sets

2016-08-22 Thread Michael Orlitzky
On 08/21/2016 03:22 PM, Damian wrote: >> > There is no such set B, as it would contain itself. The empty set contains itself.

Re: Matching infinite sets

2016-08-22 Thread Joe Quinn
On 8/21/2016 5:55 PM, Sidney Markowitz wrote: Dianne Skoll wrote on 22/08/16 8:56 AM: And... why can't a set contain itself? It can't in standard modern set theory (ZFC), through the foundation axioms, also known as the axiom of regularity https://en.wikipedia.org/wiki/Axiom_of_regularity w

Re: Matching infinite sets

2016-08-21 Thread Martin Gregorie
On Sun, 2016-08-21 at 16:56 -0400, Dianne Skoll wrote: > On Sun, 21 Aug 2016 21:22:38 +0200 > Damian wrote: > > > > > > > > > So we define set B as everything in the universe that is not in > > > set > > > A. So set B is an infinite set, everything in the universe EXCEPT > > > apples and orange

Re: Matching infinite sets

2016-08-21 Thread Sidney Markowitz
Dianne Skoll wrote on 22/08/16 8:56 AM: > And... why can't a set contain itself? > It can't in standard modern set theory (ZFC), through the foundation axioms, also known as the axiom of regularity https://en.wikipedia.org/wiki/Axiom_of_regularity which is a formulation that allows set theory t

Re: Matching infinite sets

2016-08-21 Thread Dianne Skoll
On Sun, 21 Aug 2016 21:22:38 +0200 Damian wrote: > > So we define set B as everything in the universe that is not in set > > A. So set B is an infinite set, everything in the universe EXCEPT > > apples and oranges. > There is no such set B, as it would contain itself. And... why can't a set con

Re: Matching infinite sets

2016-08-21 Thread Dianne Skoll
On Sun, 21 Aug 2016 09:47:45 -0700 Marc Perkel wrote: > So we define set B as everything in the universe that is not in set A. That's a very specific kind of infinite set. It's the complement of a finite set. Try this one on for size: Consider the set A of all positive integral powers of pi

Re: Matching infinite sets

2016-08-21 Thread Antony Stone
On Sunday 21 August 2016 at 21:22:38, Damian wrote: > Am 21.08.2016 um 18:47 schrieb Marc Perkel: > > Actually - you can match an infinite set. And maybe this is what it's > > hard for some people to wrap their head around. > > > > Suppose set A contains 2 items, apples and oranges. > > So we def

Re: Matching infinite sets

2016-08-21 Thread Damian
Am 21.08.2016 um 18:47 schrieb Marc Perkel: > Actually - you can match an infinite set. And maybe this is what it's > hard for some people to wrap their head around. > > Suppose set A contains 2 items, apples and oranges. > So we define set B as everything in the universe that is not in set A. >

Matching infinite sets

2016-08-21 Thread Marc Perkel
Actually - you can match an infinite set. And maybe this is what it's hard for some people to wrap their head around. Suppose set A contains 2 items, apples and oranges. So we define set B as everything in the universe that is not in set A. So set B is an infinite set, everything in the universe