Hi Andrew, I'm favoring Fast Serialization see here: https://github.com/RuedigerMoeller/fast-serialization
Faster than Kryo; uses regular serialize methods; can use a Unsafe Memory Map scheme; and can Serialize either to binary or json lickety split! (got Parameters & Connections serializing already, most of the work now is writing tests) won't be long! stay tuned! :-) David On Wed, Jan 6, 2016 at 11:00 AM, Andrew Dillon <[email protected]> wrote: > Thanks for the update, David! I'm glad you guys were able to sort out the > indeterminacy issue. I wonder if that was the cause of my problem, I'm > eager to test it and see. > I'm looking forward to HTM.java officially supporting serialization, > especially using those specialized APIs. > On Jan 3, 2016 10:42 AM, "cogmission (David Ray)" < > [email protected]> wrote: > >> Hi Andrew, >> >> Your issue corresponds with the current development path of HTM.java - >> and so while you might not be aware, I have been working on your issue in a >> more wholistic manner - approaching it from a standpoint of enabling >> serialization in general. The holidays slowed us down a bit, but I wanted >> to update you on what's been happening so far with this. >> >> Currently, I'm submitting a PR [1] which handles an issue which is key to >> having a predictable stable environment with which to test the serialized >> state of the algorithms; which was the indeterminacy found in successive >> executions of the TemporalMemory. That is handled with the above mentioned >> PR now. Also I've been looking into serialization schemes (API libraries >> such as Cap'N Proto, and Kryo) for the Network - though I think initially I >> will get plain old java serialization working first to gain some experience >> as to how to approach more optimized serialization via specialized >> libraries later (on an architectural level). >> >> Starting today, I will now see about getting the java serialization up >> and running and submit a PR for it. Then later this week I'll be looking >> into and starting development on a more optimized approach. >> >> Happy Holidays! >> >> David >> >> [1] https://github.com/numenta/htm.java/pull/373 >> >> >> On Mon, Dec 28, 2015 at 10:08 AM, Andrew Dillon < >> [email protected]> wrote: >> >>> Hey David, >>> >>> I attached a couple of classes that I am using in my program to manage >>> Networks and serialize/deserialize Connections. The code is a bit rough, >>> but I ran through and put in some comments to explain some things. These >>> classes are used by a GUI program to allow a user to interact with the >>> Network, but I didn't include any of the UI related stuff. Also, I've used >>> a few different kinds of data as input to the network (feeding it in using >>> the feed() method in my HTMNetwork class), but I've primarily used Sine >>> wave values for simplicity while working on the serialization and >>> deserialization aspects of the program. >>> >>> Andrew >>> >>> >>> On 12/24/2015 2:09 PM, cogmission (David Ray) wrote: >>> >>> No problem Andrew, I knew what you meant... ;-) Thanks! >>> >>> On Thu, Dec 24, 2015 at 2:07 PM, Andrew Dillon < >>> [email protected]> wrote: >>> >>>> I just realized I said I tagged them with the Serialized interface - I >>>> don't know if that interface exists - I meant the to say the Serializable >>>> interface. It's a minor point, but I don't want to cause any confusion. >>>> >>>> Thanks again, David, and Happy Holidays to you as well! >>>> On Dec 24, 2015 1:41 PM, "cogmission (David Ray)" < >>>> [email protected]> wrote: >>>> >>>>> Hi Andrew, >>>>> >>>>> Thank you for sending over the files. I will take a look and get back >>>>> to you in the next couple of days. >>>>> >>>>> Happy Holidays! >>>>> >>>>> David >>>>> >>>>> On Thu, Dec 24, 2015 at 12:46 PM, Andrew Dillon < >>>>> <[email protected]>[email protected]> wrote: >>>>> >>>>>> I attached the classes I tagged with the Serialized interface, but >>>>>> I'll list them below as well. >>>>>> >>>>>> - *Connections* >>>>>> - *Cell* >>>>>> - *Column* >>>>>> - *DistantDendrite* >>>>>> - *Pool* >>>>>> - *ProximalDendrite* >>>>>> - *Segment* >>>>>> - *Synapse* >>>>>> - *FlatMatrixSupport* >>>>>> - *SparseBinaryMatrixSupport* >>>>>> - *SparseMatrixSupport* >>>>>> - *SparseObjectMatrix* >>>>>> >>>>>> The reason I found it necessary to tag the other eleven classes >>>>>> besides *Connections *was to include all of the *Connections*'s >>>>>> members in the serialization process. Some of the classes I tagged >>>>>> are not directly used as members of *Connections*, but I needed to >>>>>> serialize them because they were superclasses of certain >>>>>> *Connections* members, and the only way to deserialize the subclass >>>>>> without serializing the superclass would have been to add a no-args >>>>>> constructor to the superclass. This would result in missing data upon >>>>>> deserialization, though, due to certain fields in the superclass not >>>>>> being >>>>>> initialized. This quick jguru article >>>>>> <http://www.jguru.com/faq/view.jsp?EID=34802> might explain what I'm >>>>>> saying better. >>>>>> >>>>>> Thanks for the assistance and no worries about a late reply. I wasn't >>>>>> even expecting a response today, since it is Christmas Eve. >>>>>> >>>>>> On 12/24/2015 11:19 AM, cogmission (David Ray) wrote: >>>>>> >>>>>> Hi Andrew, >>>>>> >>>>>> Welcome aboard! :-) >>>>>> >>>>>> First let me say thanks for using HTM.java, it's very nice to hear >>>>>> about user experiences indeed. Perhaps it would be easier if you attached >>>>>> the classes that you altered so that we can mock up our own example to >>>>>> see >>>>>> what's going on - such as the Connections.java file and whatever else was >>>>>> necessary to alter? (I would be surprised to find that you had to alter >>>>>> anything else, actually?). >>>>>> >>>>>> As this is the next thing on HTM.java's agenda, this is very >>>>>> interesting indeed... Also, as it is x-mas eve, I may not be able to get >>>>>> back to you as promptly as I otherwise would - but please send over the >>>>>> files as soon as you are able because I am anxious to play with them! ;-) >>>>>> >>>>>> Cheers, >>>>>> David >>>>>> >>>>>> On Thu, Dec 24, 2015 at 10:51 AM, Andrew Dillon < >>>>>> <[email protected]>[email protected]> wrote: >>>>>> >>>>>>> Hello all. I've been learning about HTM for a couple of months now, >>>>>>> reading On Intelligence and Numenta's papers, as well as watching their >>>>>>> videos. I just began actually working with htm.java (I'm not very >>>>>>> familiar >>>>>>> with Python, so I am unable to use that version) a week or two ago, so >>>>>>> I'm >>>>>>> no expert on it, but I have been able to create some working >>>>>>> demonstrations >>>>>>> of it. >>>>>>> >>>>>>> I have, however, run into a bit of a problem with saving networks. I >>>>>>> am aware that htm.java does not currently support this type of >>>>>>> operation ( >>>>>>> <https://github.com/numenta/htm.java/wiki/Call-To-Arms> >>>>>>> https://github.com/numenta/htm.java/wiki/Call-To-Arms), so I am >>>>>>> attempting to develop a basic method of saving and recreating my >>>>>>> networks >>>>>>> myself. What I have done thus far is modify a couple of classes to >>>>>>> implement Java's Serializable interface, in order to save my Layer's (I >>>>>>> am >>>>>>> just working with one right now) Connections object. I have succeeded in >>>>>>> serializing and deserializing the Connections object, and putting it >>>>>>> back >>>>>>> into a new Layer with the Layer.using() method. >>>>>>> >>>>>>> The problem is that when I feed the network (that is using the >>>>>>> deserialized Connections) the same data it had learned to recognize >>>>>>> before >>>>>>> I serialized it, it no longer predicts the proper values. Its output >>>>>>> looks >>>>>>> exactly like a new network; as though my saved Connections is being >>>>>>> overwritten or ignored somehow. I've spent the past few days trying to >>>>>>> figure out what is happening, digging around the source code and trying >>>>>>> a >>>>>>> few different things, but have been unable to produce any results. Do >>>>>>> any >>>>>>> of you folks have any idea how I might go about resolving this issue? >>>>>>> >>>>>>> I am sure code samples would be of interest here, but I'm not sure >>>>>>> what, specifically, I should include as my program is of a decent size. >>>>>>> If >>>>>>> anybody would like some samples, please mention what general >>>>>>> functions/areas of my program you would like to see, and I'll be happy >>>>>>> to >>>>>>> oblige. >>>>>>> >>>>>>> Thanks in advance for any help. I am very fascinated by this project >>>>>>> and HTM theory in general. I really appreciate what you all are doing >>>>>>> and >>>>>>> that this project was made open source! >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *With kind regards,* >>>>>> >>>>>> David Ray >>>>>> Java Solutions Architect >>>>>> >>>>>> *Cortical.io <http://cortical.io/>* >>>>>> Sponsor of: HTM.java <https://github.com/numenta/htm.java> >>>>>> >>>>>> [email protected] >>>>>> http://cortical.io >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *With kind regards,* >>>>> >>>>> David Ray >>>>> Java Solutions Architect >>>>> >>>>> *Cortical.io <http://cortical.io/>* >>>>> Sponsor of: HTM.java <https://github.com/numenta/htm.java> >>>>> >>>>> <[email protected]>[email protected] >>>>> <http://cortical.io/>http://cortical.io >>>>> >>>> >>> >>> >>> -- >>> *With kind regards,* >>> >>> David Ray >>> Java Solutions Architect >>> >>> *Cortical.io <http://cortical.io/>* >>> Sponsor of: HTM.java <https://github.com/numenta/htm.java> >>> >>> <[email protected]>[email protected] >>> <http://cortical.io/>http://cortical.io >>> >>> >>> >> >> >> -- >> *With kind regards,* >> >> David Ray >> Java Solutions Architect >> >> *Cortical.io <http://cortical.io/>* >> Sponsor of: HTM.java <https://github.com/numenta/htm.java> >> >> [email protected] >> http://cortical.io >> > -- *With kind regards,* David Ray Java Solutions Architect *Cortical.io <http://cortical.io/>* Sponsor of: HTM.java <https://github.com/numenta/htm.java> [email protected] http://cortical.io
