For clarity's sake, let's call the current state of NuPIC "NuPIC Alpha", and the proposed state of NuPIC (once the C++ core is extracted and we've solidified the core API) "NuPIC Beta".
Regarding the proposal that Grok continue using the legacy codebase (NuPIC Alpha) while a copied version is refactored in parallel (NuPIC Beta)... While this does allow unfettered freedom for us to create exactly the architecture we want without worrying about client contracts, it makes me uncomfortable to have Grok detached from the "development head" of the evolving NuPIC project. A solid regression test suite takes a considerable amount of time to create, and the only form of one we have right now runs within the Grok QA pipelines. Taking away the Grok dependency essentially means taking away the only regression testing NuPIC currently has. If we diverged Grok from NuPIC Beta in this fashion, I would want to get Grok pointing to the NuPIC Beta architecture ASAP, which would mean that if any unknown performance changes were introduced during the refactoring, they would be exposed only at the point when we started building Grok off NuPIC Beta. It also means that these problems might be quite hard to debug and fix, because we won't be able to point to a specific change that caused the regression failure. And the longer Grok goes without using NuPIC Beta, the more danger NuPIC Alpha is in of diverging algorithmically from NuPIC Alpha. Another reason to keep the Grok dependency is because Grok engineers might need to make algorithmic changes to NuPIC to support customer requirements for Grok, in which case they would have to make those changes on NuPIC Alpha. These changes would potentially span over the python and C++ codebases, and applying these changes to NuPIC Beta at the same time would have to be done by the NuPIC community, and it could get messy. If we apply this refactoring in incremental steps, assuring that the Grok pipeline continues to run and pass along the way, I feel like we'll end up with a stabler product in the end. I also don't think it will bind our hands to make the kinds of API changes the community wants. In fact, I think the current C++ API is rather close to fulfilling the needs of everyone. If changes to the C++ API are made, we simply update the dependent python bindings and client that Grok uses. Grok should not have to change at all. Other projects the community has created around NuPIC would also not need to change, because the python client they are using will not mutate. I don't want Grok to change in response to this refactoring (except to update the NuPIC build process due to repository and dependency changes). NuPIC already has a specific use-case with Grok, and we shouldn't break this. We can change the core C++ API all we want, as long as the Python client Grok uses doesn't change. In Summary, I don't want to detach Grok from NuPIC Alpha because: - we lose regression testing - potential of nasty algorithm merges from NuPIC Alpha by Grok engineers - the python API currently exposed to Grok by NuPIC Alpha should not change with NuPIC Beta - the extra abstraction layer of the python bindings and client provide us the flexibility to change the core API however we want --------- Matt Taylor OS Community Flag-Bearer Numenta On Fri, Jan 24, 2014 at 8:52 AM, Fergal Byrne <fergalbyrnedub...@gmail.com>wrote: > Hi Matt, > > > That email was amending its immediate predecessor, where I was suggesting > a layout for all the nupic.* repos. I'd left out nupic itself as it > currently is. > > My last mail (sent 9.46 this morning my time) supercedes this somewhat, as > it contains an alternative strategy for the extraction and future > development. Essentially what I'm saying is that a total internal fork is > the safest way to go. This leaves nupic entirely as it is today, and makes > essentially two copies - one for continuing incremental development for > Grok's purposes, using a split-out of the existing nupic into nupic/nta > (C++) and nupic.py.legacy (python); the other for radical and user-free > development of nupic.core (C++) and nupic.py (bindings, clients). > > Having to mix Grok's requirements for a stable codebase with the need to > build a real modular system creates incidental complexity which will > severely retard or prevent the success of the new development. > > This layout allows Grok to pick and choose from new improvements to both > C++ and python codebases (simply by pulling them in to nupic/nta or > nupic.py.legacy), but never to require that the new side continuously > passes all the Grok-required tests. > > Since improvements will either be to one or the other codebase, there is > no great overhead in managing the expanded list of repos. Grok never has to > compromise on what goes into nupic, and the community has the freedom to > make drastic (but planned and orderly) changes to the new architecture. > > Regards, > > Fergal Byrne > > > On Fri, Jan 24, 2014 at 3:58 PM, Matthew Taylor <m...@numenta.org> wrote: > >> On Fri, Jan 24, 2014 at 1:07 AM, Fergal Byrne < >> fergalbyrnedub...@gmail.com> wrote: >> >>> Insert at top of my list >>> >>> nupic repo remains with entire current contents >>> >>> nupic.core etc... >>> >> >> Can you explain this? I don't understand what you're saying. >> >> --------- >> Matt Taylor >> OS Community Flag-Bearer >> Numenta >> >> _______________________________________________ >> nupic mailing list >> nupic@lists.numenta.org >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> > > > -- > > Fergal Byrne, Brenter IT > > <http://www.examsupport.ie>http://inbits.com - Better Living through > Thoughtful Technology > > e:fergalbyrnedub...@gmail.com t:+353 83 4214179 > Formerly of Adnet edi...@adnet.ie http://www.adnet.ie > > _______________________________________________ > nupic mailing list > nupic@lists.numenta.org > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list nupic@lists.numenta.org http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
