If you want compare the summaries without the original text strings, then
perhaps Key is not the best approach.
This should work:
getTrigrams=: 3 ,\ ]
countTrigrams=: (~. ; #/.~)@getTrigrams
compareTrigrams=: dyad define
alltrig=. x ~.@;@,&({.) y
cnttrigs=. ((alltrig i.~ [) { 0 ,~ ])&>/&> x;<y
alltrig ; |: cnttrigs
)
]'X Y'=: 27 ({. ; }.) 'actg' {~ 60 ?@$ 4
]Xsumry=: countTrigrams X
]Ysumry=: countTrigrams Y
Xsumry compareTrigrams Ysumry
On Sat, Oct 12, 2019 at 6:51 PM 'Jim Russell' via Programming <
[email protected]> wrote:
> Looks promising. Typically, the strings are different lengths, and we may
> not have access to them at the same time. (Which is why I hade the
> intermediate summary step.) Let me ponder that (I don't think it will
> matter) while I study your approach more. Thanks very much!
>
> > On Oct 12, 2019, at 1:22 AM, Ric Sherlock <[email protected]> wrote:
> >
> > Here's one approach...
> >
> > I find it much easier to work with if there is actual data. The following
> > may not be representative of your data but it gives us somewhere to
> start.
> >
> > ]'X Y'=: 'actg' {~ 2 30 ?@$ 4
> >
> > ggtaaaatgactgtagtgaagaaggagtcc
> >
> > ctgattaaggttcggtgtcgataccgcgca
> >
> >
> > We now have 2 strings X and Y. Let's obtain the trigrams for each string
> >
> > trig=: 3,\&.> X;Y Get the nub of the union of both sets of trigrams and
> > prepend it to each trigram set. supertrig=: (,~&.> <@~.@;) trig Now we
> can
> > use Key to count the trigrams in each set and decrement by 1 (for the
> extra
> > copy that we added). <: #/.~&> supertrig
> >
> > 1 2 1 2 1 1 2 1 1 1 1 1 2 1 2 2 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
> >
> > 2 0 1 0 0 0 1 0 0 1 1 0 0 1 0 1 0 1 0 0 1 0 2 1 1 1 1 2 1 1 1 1 1 1 2 1 1
> >
> > Or to summarise by trigram:
> >
> > (~.@; trig);|: <: #/.~&> supertrig
> >
> > +---+---+
> >
> > |ggt|1 2|
> >
> > |gta|2 0|
> >
> > |taa|1 1|
> >
> > |aaa|2 0|
> >
> > |aat|1 0|
> >
> > |atg|1 0|
> >
> > |tga|2 1|
> >
> > |gac|1 0|
> >
> > |act|1 0|
> >
> > |ctg|1 1|
> >
> > |tgt|1 1|
> >
> > |tag|1 0|
> >
> > |agt|2 0|
> >
> > |gtg|1 1|
> >
> > |gaa|2 0|
> >
> > |aag|2 1|
> >
> > |aga|1 0|
> >
> > |agg|1 1|
> >
> > |gga|1 0|
> >
> > |gag|1 0|
> >
> > |gtc|1 1|
> >
> > |tcc|1 0|
> >
> > |gat|0 2|
> >
> > |att|0 1|
> >
> > |tta|0 1|
> >
> > |gtt|0 1|
> >
> > |ttc|0 1|
> >
> > |tcg|0 2|
> >
> > |cgg|0 1|
> >
> > |cga|0 1|
> >
> > |ata|0 1|
> >
> > |tac|0 1|
> >
> > |acc|0 1|
> >
> > |ccg|0 1|
> >
> > |cgc|0 2|
> >
> > |gcg|0 1|
> >
> > |gca|0 1|
> >
> > +---+---+
> >
> >
> >> On Sat, Oct 12, 2019 at 4:40 PM 'Jim Russell' via Programming <
> >> [email protected]> wrote:
> >>
> >> Sure, thanks. I'm working to re-implement a text comparison program I
> did
> >> using VBA & Microsoft Access a number of years back.
> >>
> >> The object is to compare two text documents and see how similar one is
> to
> >> the other by comparing the number of unique trigrams that are found in
> >> each.
> >> For each text string a table of trigrams is constructed with the
> >> expression 3,\x. The resulting table of 3-character samples m is then
> >> tallied using #/.~m . This yields a vector of counts of each unique
> trigram
> >> corresponding to (an unseen) nub of m. The count, and a copy of the nub
> of
> >> m, represent a summary of the text in string x.
> >> This same process then repeated to creat a smry for the second string,
> y.
> >>
> >> The next step in the process is to assign a score of 0 to 1 based on a
> >> comparison of the two string summaries. It would seem sensible to
> compare
> >> the nub of the two text strings to each other. What is the difference in
> >> counts between the trigrams they have in common, and how many trigram
> hits
> >> for each are unique?
> >> That is where using nub1 #/. nub2 would be attractive, were it not
> >> required that the arguments had the same row counts, and Key could not
> >> count unmatched rows.
> >>
> >> As it stands, I fear I am duplicating effort to find the nubs in
> preparing
> >> the summaries, and again if I have to use i. to calculate the scores.
> If I
> >> get a vector result when I use key on vectors, might I expect a table
> >> result (including the counts and the nub) when key is applied to tables?
> >>
> >> Or is there a more appropriate approach? (In access and VBA, I used
> >> dictionary objects with 3 character keys, as I recall. But I was very
> >> pleasantly surprised at how well the 3 character trigrams recognized
> text
> >> similarities.)
> >>
> >> I really appreciate any insights you might have, Ric, and thanks for
> >> tolerating my ignorance.
> >>
> >>>> On Oct 11, 2019, at 10:23 PM, Ric Sherlock <[email protected]> wrote:
> >>>
> >>> Not sure I'm understanding your questions. Maybe including some of the
> >>> expressions you've tried to illustrate your points would help?
> >>
> >> ----------------------------------------------------------------------
> >> For information about J forums see http://www.jsoftware.com/forums.htm
> >>
> > ----------------------------------------------------------------------
> > For information about J forums see http://www.jsoftware.com/forums.htm
>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
>
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
