Re: [opencog-dev] Pattern Mining and memory issues

'Nil Geisweiller' via opencog Thu, 03 Jan 2019 12:54:19 -0800

On 1/3/19 9:01 PM, Johannes Castner wrote:

Actually, I couldn't wait to try it and I found that it seems to beworking!


Cool!

Does it still save files with the mined patterns, or does itjust store those mined patterns in the Atom Space?

The list of discovered patterns with minimum support is directlyreturned by cog-mine, and a fortiori stored in the atomspace. If thatlist is empty it means no pattern with minimum support were found, whichprobably indicates that you need to lower the minimum support.


Carefully read

(help cog-mine)

and if not enough the README.md which should provide you the backgroundknowledge you need to understand the help.

Nil


Johannes

On Thursday, January 3, 2019 at 6:27:31 PM UTC, Nil wrote:

    Hi Johannes,

    On 1/3/19 4:36 PM, Johannes Castner wrote:
     > Hi Nil,
     >
     > I thought to adopt the one for the GO.scm as in this example?
     > https://github.com/opencog/test-datasets/tree/master/go
    <https://github.com/opencog/test-datasets/tree/master/go>

    It seems that code is more recent

    https://github.com/opencog/agi-bio/tree/master/knowledge-import
    <https://github.com/opencog/agi-bio/tree/master/knowledge-import>

     > I have not yet seriously worked on it as I need to first be sure
    that
     > the patternminer works on my system and I haven't been able to get
     > something very simple to work as expected yet.  I'll be tinkering
    again \

    I should mention that the new pattern miner is already quite stable. If
    you can install opencog on your system, it should work.

     > ...Of course I'd love to share the results and what I'm finding with
     > you; it'd be great if I can in some ways contribute to your awesome

    It would also be of interest to the AGI-Bio team.

     > system! Does my plan sound reasonable?  In terms of timing; I really
     > need to have a working paper using this system ready before April
    1st,
     > so that I can distribute my dissertation to my committee.  This
    means
     > that I'd love to have the results as soon as possible; preferably
    before
     > the end of January if that is possible?

    That seems doable. I should start implementing surprisingness next
    week.
    The time it takes depends on how it is implemented, I'm aiming at
    something as general as possible but might also start with something
    more specialized, I mean I'll do my best so you have something that
    works on time.

    Meanwhile if you can run the pattern miner on your data just to make
    sure it works so far, maybe even verify at hand that it can find
    surprising patterns, etc, would be very good.

    Nil

     >
     > Johannes
     >
     > On Thursday, January 3, 2019 at 9:52:09 AM UTC, Nil wrote:
     >
     >     On 1/3/19 11:37 AM, Johannes Castner wrote:
     >      > Hi Nil,
     >      >
     >      > Does the new pattern miner support surprisingness yet?
     >
     >     Nope :-(
     >
     >     That's the last missing bit really.
     >
     >      > That's what I'm hoping to make use of.
     >
     >     Great.
     >
     >     I might have recommended you to use the old pattern miner in the
     >     meantime but I'm afraid it's gonna be too buggy. And
    implementing
     >     surprisingness in the new pattern miner is pretty much my
    next task,
     >     so...
     >
     >     What is your timing? When do you need this?
     >
     >     If I understand correctly you want to mine patterns in
     >
     >
    https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo
    <https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    
<https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>>

     >
     >
     >     and I suppose you already have some scripts to turn that into
    atoms.
     >     Could you share that script? we might want to add it to
     >
     > https://github.com/opencog/external-tools
    <https://github.com/opencog/external-tools>
     >     <https://github.com/opencog/external-tools
    <https://github.com/opencog/external-tools>>
     >
     >     and maybe turn your use-case into a pattern miner example, that
     >     would be
     >     great.
     >
     >     Nil
     >
     >      >
     >      > Johannes
     >      >
     >      > On Thursday, January 3, 2019 at 9:07:25 AM UTC, Nil wrote:
     >      >
     >      >     Hi,
     >      >
     >      >     you should use the new pattern miner, see
     >      >
     >      >
     >
    
https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md
    
<https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md>

><https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md

    
<https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md>>

     >
     >      >

><https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md

    
<https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md>

><https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md

    
<https://github.com/opencog/opencog/blob/master/opencog/learning/miner/README.md>>>

     >
     >      >
     >      >
     >      >     BTW, what doc did you follow to get to that command? I'm
     >     asking so I
     >      >     can
     >      >     redirect the user to the new pattern miner.
     >      >
     >      >     Nil
     >      >
     >      >     On 1/1/19 3:57 PM, Johannes Castner wrote:
     >      >      > Hi Nil,
     >      >      >
     >      >      > Thank you, that did that trick.  Now I run:
     >      >      >
     >      >      > guile> (pm-run-patternminer)
     >      >      >
     >      >      > Yet no matter what knowledge base I run this on it
    never
     >     seems to
     >      >     find any interesting patterns. What could be the cause
    of that?
     >      >      >
     >      >      > Johannes
     >      >      >
     >      >      > On Sunday, December 30, 2018 at 9:43:13 PM UTC, Nil
    wrote:
     >      >      >
     >      >      >     Hi Johannes,
     >      >      >
     >      >      >     I'm not sure what you mean by compilation but
    it is
     >      >     recommended to
     >      >      >     disable guile compilation as it makes loading
    large
     >     data set
     >      >      >     impractical. For that do as follows
     >      >      >
     >      >      >     guile --no-auto-compile -l my-data.scm
     >      >      >
     >      >      >     where my-data.scm is the file containing your
     >     atomspace data.
     >      >      >
     >      >      >     Hope it helps, let me know otherwise.
     >      >      >
     >      >      >     Nil
     >      >      >
     >      >      >     On 12/30/18 3:02 PM, Johannes Castner wrote:
     >      >      >      > I'm quite interested in using the pattern
    miner for
     >     the last
     >      >      >     chapter of
     >      >      >      > my dissertation and so I'm excited to work
    with it
     >     and try
     >      >     things
     >      >      >     out.
     >      >      >      > However, it seems as though the compilation
    of even
     >     small
     >      >     test
     >      >      >     knowledge
     >      >      >      > bases takes a lot of memory?  For serious data
     >     sets, such
     >      >     as the
     >      >      >      > environmental ontology
     >      >      >      >
     >      >      >
     >      >

>(https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    <https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    
<https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>>

     >
     >      >

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    <https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    
<https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>>>

     >
     >      >
     >      >      >
     >      >

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    <https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    
<https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>>

     >
     >      >

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    <https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>

><https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo

    
<https://raw.githubusercontent.com/EnvironmentOntology/envo/master/envo.obo>>>>)

     >
     >      >
     >      >      >
     >      >      >      > it always crashes.  Is there a way to
    compile the
     >     data in
     >      >     a more
     >      >      >     memory
     >      >      >      > efficient way, or compile it somehow
    elsewhere and
     >     upload it?
     >      >      >      >
     >      >      >      > --
     >      >      >      > You received this message because you are
     >     subscribed to
     >      >     the Google
     >      >      >      > Groups "opencog" group.
     >      >      >      > To unsubscribe from this group and stop
    receiving
     >     emails
     >      >     from it,
     >      >      >     send
     >      >      >      > an email to [email protected]
     >     <javascript:>
     >      >      >      > <mailto:[email protected]
    <javascript:>>.
     >      >      >      > To post to this group, send email to
     > [email protected]
     >      >      >     <javascript:>
     >      >      >      > <mailto:[email protected] <javascript:>>.
     >      >      >      > Visit this group at
     >      > https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>
     >     <https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>>
     >      >     <https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>
     >     <https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>>>
     >      >      >     <https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>
     >     <https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>>
     >      >     <https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>
     >     <https://groups.google.com/group/opencog
    <https://groups.google.com/group/opencog>>>>.
     >      >      >      > To view this discussion on the web visit
     >      >      >      >
     >      >      >
     >      >
     >
    
https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com
    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>

><https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com

    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>>

     >
     >      >

><https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com

    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>

><https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com

    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>>>

     >
     >      >
     >      >      >
     >      >

><https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com

    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>

><https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com

    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>>

     >
     >      >

><https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com

    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>

><https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com

    
<https://groups.google.com/d/msgid/opencog/da7f4204-8b1d-47d8-8cc1-0d05da2048e8%40googlegroups.com>>>>

     >
     >      >
     >      >      >
     >      >      >      >
     >      >      >
     >      >