Thanks. (Except for the part about etaionshrdlu... and the fact that I was
unaware of any difference between prepend and prefix.) I had considered texts
using different character sets, and figured I would be comfortable reporting
them as completely different.
Thanks for the insights!
(I should credit Skip Cave for first mentioning trigrams in this form long ago.)
> On Oct 12, 2019, at 12:59 PM, 'Mike Day' via Programming
> <programm...@jsoftware.com> wrote:
>
> The advantage is that you’re in control of the domain of interest. You know
> all cases that _might_ arise.
>
> With my example of a given alphabet, we might be looking at the frequency of
> letters in English prose, and find their sort order is along the lines of
> etaionshrdlu... (iirc).
>
> Not so good if you come across kanji, say, assuming it’s something that’s
> foreign to you, and you don’t know the alphabet/symbol set. Then you can
> only work on the symbols, cases, that you encounter.
>
> Even then, though, you can compare two sets by using the nub of their union
> as the basis for frequency analysis. In that case, I would prepend (not
> prefix, pardon my slip earlier) that nub to each series, and, once again,
> decrement all counts for each series by one.
>
> Any clearer?
>
> Cheers,
>
> Mike
>
>
>
> Sent from my iPad
>
>> On 12 Oct 2019, at 16:26, 'Jim Russell' via Programming
>> <programm...@jsoftware.com> wrote:
>>
>> Thanks, interesting. What advantage that give you?
>>
>>>> On Oct 12, 2019, at 9:02 AM, 'Mike Day' via Programming
>>>> <programm...@jsoftware.com> wrote:
>>>
>>> A quick thought, might not be what you have in mind.
>>>
>>> If, say, you’re seeking the frequency of letters, it’s worth prefixing the
>>> sorted alphabet of interest to your string and then subtracting one from
>>> the scores.
>>>
>>> Useful for me sometimes, anyway.
>>>
>>> Mike
>>>
>>> Sent from my iPad
>>>
>>>> On 12 Oct 2019, at 06:50, 'Jim Russell' via Programming
>>>> <programm...@jsoftware.com> wrote:
>>>>
>>>> Looks promising. Typically, the strings are different lengths, and we may
>>>> not have access to them at the same time. (Which is why I hade the
>>>> intermediate summary step.) Let me ponder that (I don't think it will
>>>> matter) while I study your approach more. Thanks very much!
>>>>
>>>>>> On Oct 12, 2019, at 1:22 AM, Ric Sherlock <tikk...@gmail.com> wrote:
>>>>>
>>>>> Here's one approach...
>>>>>
>>>>> I find it much easier to work with if there is actual data. The following
>>>>> may not be representative of your data but it gives us somewhere to start.
>>>>>
>>>>> ]'X Y'=: 'actg' {~ 2 30 ?@$ 4
>>>>>
>>>>> ggtaaaatgactgtagtgaagaaggagtcc
>>>>>
>>>>> ctgattaaggttcggtgtcgataccgcgca
>>>>>
>>>>>
>>>>> We now have 2 strings X and Y. Let's obtain the trigrams for each string
>>>>>
>>>>> trig=: 3,\&.> X;Y Get the nub of the union of both sets of trigrams and
>>>>> prepend it to each trigram set. supertrig=: (,~&.> <@~.@;) trig Now we can
>>>>> use Key to count the trigrams in each set and decrement by 1 (for the
>>>>> extra
>>>>> copy that we added). <: #/.~&> supertrig
>>>>>
>>>>> 1 2 1 2 1 1 2 1 1 1 1 1 2 1 2 2 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>>>
>>>>> 2 0 1 0 0 0 1 0 0 1 1 0 0 1 0 1 0 1 0 0 1 0 2 1 1 1 1 2 1 1 1 1 1 1 2 1 1
>>>>>
>>>>> Or to summarise by trigram:
>>>>>
>>>>> (~.@; trig);|: <: #/.~&> supertrig
>>>>>
>>>>> +---+---+
>>>>>
>>>>> |ggt|1 2|
>>>>>
>>>>> |gta|2 0|
>>>>>
>>>>> |taa|1 1|
>>>>>
>>>>> |aaa|2 0|
>>>>>
>>>>> |aat|1 0|
>>>>>
>>>>> |atg|1 0|
>>>>>
>>>>> |tga|2 1|
>>>>>
>>>>> |gac|1 0|
>>>>>
>>>>> |act|1 0|
>>>>>
>>>>> |ctg|1 1|
>>>>>
>>>>> |tgt|1 1|
>>>>>
>>>>> |tag|1 0|
>>>>>
>>>>> |agt|2 0|
>>>>>
>>>>> |gtg|1 1|
>>>>>
>>>>> |gaa|2 0|
>>>>>
>>>>> |aag|2 1|
>>>>>
>>>>> |aga|1 0|
>>>>>
>>>>> |agg|1 1|
>>>>>
>>>>> |gga|1 0|
>>>>>
>>>>> |gag|1 0|
>>>>>
>>>>> |gtc|1 1|
>>>>>
>>>>> |tcc|1 0|
>>>>>
>>>>> |gat|0 2|
>>>>>
>>>>> |att|0 1|
>>>>>
>>>>> |tta|0 1|
>>>>>
>>>>> |gtt|0 1|
>>>>>
>>>>> |ttc|0 1|
>>>>>
>>>>> |tcg|0 2|
>>>>>
>>>>> |cgg|0 1|
>>>>>
>>>>> |cga|0 1|
>>>>>
>>>>> |ata|0 1|
>>>>>
>>>>> |tac|0 1|
>>>>>
>>>>> |acc|0 1|
>>>>>
>>>>> |ccg|0 1|
>>>>>
>>>>> |cgc|0 2|
>>>>>
>>>>> |gcg|0 1|
>>>>>
>>>>> |gca|0 1|
>>>>>
>>>>> +---+---+
>>>>>
>>>>>
>>>>>> On Sat, Oct 12, 2019 at 4:40 PM 'Jim Russell' via Programming <
>>>>>> programm...@jsoftware.com> wrote:
>>>>>>
>>>>>> Sure, thanks. I'm working to re-implement a text comparison program I did
>>>>>> using VBA & Microsoft Access a number of years back.
>>>>>>
>>>>>> The object is to compare two text documents and see how similar one is to
>>>>>> the other by comparing the number of unique trigrams that are found in
>>>>>> each.
>>>>>> For each text string a table of trigrams is constructed with the
>>>>>> expression 3,\x. The resulting table of 3-character samples m is then
>>>>>> tallied using #/.~m . This yields a vector of counts of each unique
>>>>>> trigram
>>>>>> corresponding to (an unseen) nub of m. The count, and a copy of the nub
>>>>>> of
>>>>>> m, represent a summary of the text in string x.
>>>>>> This same process then repeated to creat a smry for the second string, y.
>>>>>>
>>>>>> The next step in the process is to assign a score of 0 to 1 based on a
>>>>>> comparison of the two string summaries. It would seem sensible to compare
>>>>>> the nub of the two text strings to each other. What is the difference in
>>>>>> counts between the trigrams they have in common, and how many trigram
>>>>>> hits
>>>>>> for each are unique?
>>>>>> That is where using nub1 #/. nub2 would be attractive, were it not
>>>>>> required that the arguments had the same row counts, and Key could not
>>>>>> count unmatched rows.
>>>>>>
>>>>>> As it stands, I fear I am duplicating effort to find the nubs in
>>>>>> preparing
>>>>>> the summaries, and again if I have to use i. to calculate the scores. If
>>>>>> I
>>>>>> get a vector result when I use key on vectors, might I expect a table
>>>>>> result (including the counts and the nub) when key is applied to tables?
>>>>>>
>>>>>> Or is there a more appropriate approach? (In access and VBA, I used
>>>>>> dictionary objects with 3 character keys, as I recall. But I was very
>>>>>> pleasantly surprised at how well the 3 character trigrams recognized text
>>>>>> similarities.)
>>>>>>
>>>>>> I really appreciate any insights you might have, Ric, and thanks for
>>>>>> tolerating my ignorance.
>>>>>>
>>>>>>>> On Oct 11, 2019, at 10:23 PM, Ric Sherlock <tikk...@gmail.com> wrote:
>>>>>>>
>>>>>>> Not sure I'm understanding your questions. Maybe including some of the
>>>>>>> expressions you've tried to illustrate your points would help?
>>>>>>
>>>>>> ----------------------------------------------------------------------
>>>>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>>>>>
>>>>> ----------------------------------------------------------------------
>>>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>>>
>>>> ----------------------------------------------------------------------
>>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>> ----------------------------------------------------------------------
>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
