Rik, We decided to have this conversation on the nupic-discuss list first, because it will affect how users build and install NuPIC. Once we've made some decisions, we can take the implementation discussion to nupic-hackers (anyone is welcome to do this). --------- Matt Taylor OS Community Flag-Bearer Numenta
On Fri, Apr 11, 2014 at 9:38 PM, Rik <[email protected]> wrote: > Probably too late at this point but this thread go to nupic-hackers? > > On Sat, Apr 12, 2014, at 08:36 AM, Subutai Ahmad wrote: >> Yes, I think this is common. There are many Python packages that also >> compile C++ libraries. In those cases setup.py calls make, but in our >> case >> we can have it call cmake. This means all the current work can stay >> pretty >> much as-is. >> >> --Subutai >> >> >> On Fri, Apr 11, 2014 at 3:30 PM, Chetan Surpur <[email protected]> >> wrote: >> >> > I think setup.py should call cmake, since Python is a layer on top of C++, >> > and C++ / cmake should be independent of Python. >> > >> > >> > On Fri, Apr 11, 2014 at 3:23 PM, David Ragazzi >> > <[email protected]>wrote: >> > >> >> > In my ideal world, I think we start with cmake and make to build the >> >> c++ components, and then setup.py for the python installation in lieu of a >> >> typical `make install`, but both would be required: >> >> > (cmake $NUPIC && make) >> >> python setup.py (develop|install) OR easy_install . OR pip install . OR >> >> manual PYTHONPATH fixup. >> >> >> >> This is really a cruel question... >> >> >> >> As NuPIC is a mixture of Python and C++ code, I don't know surely if >> >> CMake should call setup.py or if setup.py should call CMake.. :-( >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 11 April 2014 18:57, Austin Marshall <[email protected]> wrote: >> >> >> >>> The problem with standards is there are so many! The layout you've >> >>> drawn here is fairly typical and consistent with what I would expect. In >> >>> short, there's usually a root directory with a README and setup.py, and >> >>> maybe bin/ docs/ and any number of python packages, but usually just one. >> >>> The term "package" is meaningful within the python ecosystem -- it's any >> >>> directory with a file called __init__.py, which can be empty, and usually >> >>> contains additional python modules (files with a .py suffix) or packages. >> >>> >> >>> Where it gets difficult is that there are multiple competing packaging >> >>> implementations (setuptools, distutils, etc), multiple competing >> >>> packaging >> >>> formats (egg vs wheel, binary vs source), multiple competing installers >> >>> (easy_install vs pip vs setup.py), and then of course the issue of c/c++ >> >>> extensions, all of which may be interchangeable at various different >> >>> stages. For many of the approaches there is some redundancy with the way >> >>> we're using cmake with NuPIC. >> >>> >> >>> In my ideal world, I think we start with cmake and make to build the c++ >> >>> components, and then setup.py for the python installation in lieu of a >> >>> typical `make install`, but both would be required: >> >>> >> >>> (cmake $NUPIC && make) >> >>> python setup.py (develop|install) OR easy_install . OR pip install . OR >> >>> manual PYTHONPATH fixup. >> >>> >> >>> The cmake and make commands build in-place, don't require env variables, >> >>> and doesn't affect anything outside of the $NUPIC source tree. Then, the >> >>> python setup.py ecosystem is used for using/installing the final python >> >>> package. I think this approach gives maximum control and flexibility. >> >>> >> >>> >> >>> >> >>> On Fri, Apr 11, 2014 at 2:42 PM, David Ragazzi >> >>> <[email protected]>wrote: >> >>> >> >>>> Ok.. I found this one and liked it.. It also seems being well >> >>>> recommended for Python projects.. >> >>>> >> >>>> I adapted to NuPIC.. >> >>>> >> >>>> *Nupic/* >> >>>> *|-- bin/* >> >>>> *| |-- nupic* >> >>>> *|* >> >>>> *|-- docs/* >> >>>> *|* >> >>>> *|-- nupic/* >> >>>> *| |-- test/* >> >>>> *| | |-- __init__.py* >> >>>> *| | |-- test_main.py* >> >>>> *| | * >> >>>> *| |-- __init__.py* >> >>>> *| |-- main.py* >> >>>> *|* >> >>>> *|-- setup.py* >> >>>> *|-- README.md* >> >>>> >> >>>> I got this model from here: >> >>>> >> >>>> http://stackoverflow.com/questions/193161/what-is-the-best-project-structure-for-a-python-application >> >>>> >> >>>> >> >>>> Which is inspired in best practices recommended here: >> >>>> >> >>>> http://as.ynchrono.us/2007/12/filesystem-structure-of-python-project_21.html >> >>>> >> >>>> Subutai, Chetan, Fergal, and others: what do you think? It has setup.py >> >>>> in root, the binaries and source are in separated locations, etc.. >> >>>> However, >> >>>> we could improve it... >> >>>> >> >>>> David >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On 11 April 2014 16:07, David Ragazzi <[email protected]> wrote: >> >>>> >> >>>>> Sorry for my ignorancy.. But which is default structure for Python >> >>>>> projects? Please,someone explain to me in order to we can align what >> >>>>> we >> >>>>> want and the current process does.. >> >>>>> >> >>>>> >> >>>>> On 11 April 2014 15:53, Fergal Byrne >> >>>>> <[email protected]>wrote: >> >>>>> >> >>>>>> +1 on Subutai's view. IMHO it is a *requirement* for NuPIC to be like >> >>>>>> any other OSS software. >> >>>>>> >> >>>>>> For nupic.core, we should have cd build; cmake ..; make; [make test;] >> >>>>>> make install (installs to std locations /usr/local/bin, lib, include) >> >>>>>> For nupic.py; setup.py or pip install (installs to same place as >> >>>>>> everything else) >> >>>>>> >> >>>>>> There should be no environment variables to remember or set. >> >>>>>> >> >>>>>> This will require serious reorganisation, but it is the price of >> >>>>>> getting NuPIC out in the wild. >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Fri, Apr 11, 2014 at 7:27 PM, Matthew Taylor >> >>>>>> <[email protected]>wrote: >> >>>>>> >> >>>>>>> I can see Subutai's point here. While nupic.core is a C++ project, >> >>>>>>> nupic is a python project. We should try to maintain python standards >> >>>>>>> as much as possible, even though we have to deal with a C++ build as >> >>>>>>> well. >> >>>>>>> --------- >> >>>>>>> Matt Taylor >> >>>>>>> OS Community Flag-Bearer >> >>>>>>> Numenta >> >>>>>>> >> >>>>>>> >> >>>>>>> On Fri, Apr 11, 2014 at 11:18 AM, Subutai Ahmad <[email protected]> >> >>>>>>> wrote: >> >>>>>>> > >> >>>>>>> > I'm not an expert in this. I just know it is really really hard to >> >>>>>>> make >> >>>>>>> > these things robust in all situations. For example, below is my >> >>>>>>> sys.path >> >>>>>>> > - where would you install nupic? >> >>>>>>> > >> >>>>>>> > I strongly recommend we use the standard python way of doing this, >> >>>>>>> whatever >> >>>>>>> > that is. >> >>>>>>> > >> >>>>>>> > --Subutai >> >>>>>>> > >> >>>>>>> > ['', >> >>>>>>> > '/Users/Subutai/local/bin', >> >>>>>>> > '/Users/Subutai/nta/grok-projects/gef', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/Users/Subutai/local/lib/python2.7/site-packages/pandas-0.13.1-py2.7-macosx-10.9-intel.egg', >> >>>>>>> > >> >>>>>>> >> >>>>>>> '/Users/Subutai/local/lib/python2.7/site-packages/plotly-0.5.7-py2.7.egg', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/Users/Subutai/local/lib/python2.7/site-packages/simplejson-3.4.0-py2.7.egg', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/Users/Subutai/nta/nupic-darwin64/lib/python2.7/site-packages/setuptools-1.0-py2.7.egg', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/Users/Subutai/nta/nupic-darwin64/lib/python2.7/site-packages/pip-1.4.1-py2.7.egg', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/Users/Subutai/Library/Python/2.7/lib/python/site-packages/setuptools-2.0.1-py2.7.egg', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/Library/Python/2.7/site-packages/readline-6.2.4.1-py2.7-macosx-10.7-intel.egg', >> >>>>>>> > >> >>>>>>> >> >>>>>>> '/Users/Subutai/nta/nupic-darwin64/nta/eng/lib/python2.7/site-packages', >> >>>>>>> > '/Users/Subutai/nta/grok-projects', >> >>>>>>> > '/Users/Subutai/local/lib/python2.7/site-packages', >> >>>>>>> > '/Users/Subutai/nta/grok', >> >>>>>>> > '/Users/Subutai/nta/nupic-darwin64/lib64/python2.7/site-packages', >> >>>>>>> > '/Users/Subutai/nta/nupic-darwin64/lib/python2.7/site-packages', >> >>>>>>> > '/Users/Subutai/local/src/ipython-1.1.0', >> >>>>>>> > >> >>>>>>> '/Users/Subutai/nta/nupic-darwin64/lib/python2.7/site-packages/PIL', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', >> >>>>>>> > >> >>>>>>> >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', >> >>>>>>> > '/Users/Subutai/Library/Python/2.7/lib/python/site-packages', >> >>>>>>> > >> >>>>>>> > >> >>>>>>> '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', >> >>>>>>> > '/Library/Python/2.7/site-packages', >> >>>>>>> > '/Users/Subutai/nta/nupic-darwin64/lib64/python2.7/site-packages', >> >>>>>>> > >> >>>>>>> >> >>>>>>> '/Users/Subutai/local/lib/python2.7/site-packages/IPython/extensions'] >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > On Fri, Apr 11, 2014 at 11:12 AM, David Ragazzi < >> >>>>>>> [email protected]> >> >>>>>>> > wrote: >> >>>>>>> >> >> >>>>>>> >> We could use this in CMake: >> >>>>>>> >> >> >>>>>>> >> execute_process(COMMAND ${python} import sys; print sys.path; >> >>>>>>> >> OUTPUT_VARIABLE PYTHON_PATH) >> >>>>>>> >> >> >>>>>>> >> and then install python libraries to ${PYTHON_PATH}/nupic >> >>>>>>> >> >> >>>>>>> >> :-) >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> On 11 April 2014 15:06, David Ragazzi <[email protected]> >> >>>>>>> wrote: >> >>>>>>> >>> >> >>>>>>> >>> Subutai, >> >>>>>>> >>> >> >>>>>>> >>> Isn't.. sys.path py function return default location >> >>>>>>> (PYTHONPATH).. It's >> >>>>>>> >>> just we use this default location! >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> On 11 April 2014 15:01, Subutai Ahmad <[email protected]> >> >>>>>>> wrote: >> >>>>>>> >>>> >> >>>>>>> >>>> >> >>>>>>> >>>> Yes, but that *requires* you to set an environment variable. >> >>>>>>> >>>> >> >>>>>>> >>>> --Subutai >> >>>>>>> >>>> >> >>>>>>> >>>> On Fri, Apr 11, 2014 at 10:16 AM, Matthew Taylor < >> >>>>>>> [email protected]> >> >>>>>>> >>>> wrote: >> >>>>>>> >>>>> >> >>>>>>> >>>>> On Fri, Apr 11, 2014 at 9:58 AM, Subutai Ahmad < >> >>>>>>> [email protected]> >> >>>>>>> >>>>> wrote: >> >>>>>>> >>>>> > To see why it would not work consider this: I should be able >> >>>>>>> to login >> >>>>>>> >>>>> > to my >> >>>>>>> >>>>> > home directory, type python, and then do "import nupic". It >> >>>>>>> is >> >>>>>>> >>>>> > necessary to >> >>>>>>> >>>>> > support this. Unless the NuPIC release is *placed* in a >> >>>>>>> standard >> >>>>>>> >>>>> > location, >> >>>>>>> >>>>> > python will have no idea where nupic is. >> >>>>>>> >>>>> >> >>>>>>> >>>>> +1. I do this all the time. However, I can easily update my >> >>>>>>> PYTHONPATH >> >>>>>>> >>>>> after I've built NuPIC. >> >>>>>>> >>>>> >> >>>>>>> >>>>> >> >>>>>>> >>>>> >> >>>>>>> >>>>> --------- >> >>>>>>> >>>>> Matt Taylor >> >>>>>>> >>>>> OS Community Flag-Bearer >> >>>>>>> >>>>> Numenta >> >>>>>>> >>>>> >> >>>>>>> >>>>> _______________________________________________ >> >>>>>>> >>>>> nupic mailing list >> >>>>>>> >>>>> [email protected] >> >>>>>>> >>>>> >> >>>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>>>>>> >>>> >> >>>>>>> >>>> >> >>>>>>> >>>> >> >>>>>>> >>>> _______________________________________________ >> >>>>>>> >>>> nupic mailing list >> >>>>>>> >>>> [email protected] >> >>>>>>> >>>> >> >>>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>>>>>> >>>> >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> >> >>>>>>> >>> -- >> >>>>>>> >>> David Ragazzi >> >>>>>>> >>> OS Community Commiter >> >>>>>>> >>> Numenta.org >> >>>>>>> >>> -- >> >>>>>>> >>> "I think James Connolly, the Irish revolutionary, is right when >> >>>>>>> he says >> >>>>>>> >>> that the only prophets are those who make their future. So we're >> >>>>>>> not >> >>>>>>> >>> anticipating, we're working for it." >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> -- >> >>>>>>> >> David Ragazzi >> >>>>>>> >> OS Community Commiter >> >>>>>>> >> Numenta.org >> >>>>>>> >> -- >> >>>>>>> >> "I think James Connolly, the Irish revolutionary, is right when >> >>>>>>> he says >> >>>>>>> >> that the only prophets are those who make their future. So we're >> >>>>>>> not >> >>>>>>> >> anticipating, we're working for it." >> >>>>>>> >> >> >>>>>>> >> _______________________________________________ >> >>>>>>> >> nupic mailing list >> >>>>>>> >> [email protected] >> >>>>>>> >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>>>>>> >> >> >>>>>>> > >> >>>>>>> > >> >>>>>>> > _______________________________________________ >> >>>>>>> > nupic mailing list >> >>>>>>> > [email protected] >> >>>>>>> > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>>>>>> > >> >>>>>>> >> >>>>>>> _______________________________________________ >> >>>>>>> nupic mailing list >> >>>>>>> [email protected] >> >>>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> -- >> >>>>>> >> >>>>>> Fergal Byrne, Brenter IT >> >>>>>> >> >>>>>> Author, Real Machine Intelligence with Clortex and NuPIC >> >>>>>> https://leanpub.com/realsmartmachines >> >>>>>> >> >>>>>> <http://www.examsupport.ie>http://inbits.com - Better Living through >> >>>>>> Thoughtful Technology >> >>>>>> http://ie.linkedin.com/in/fergbyrne/ >> >>>>>> https://github.com/fergalbyrne >> >>>>>> >> >>>>>> e:[email protected] t:+353 83 4214179 >> >>>>>> Formerly of Adnet [email protected] http://www.adnet.ie >> >>>>>> >> >>>>>> _______________________________________________ >> >>>>>> nupic mailing list >> >>>>>> [email protected] >> >>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> David Ragazzi >> >>>>> OS Community Commiter >> >>>>> Numenta.org >> >>>>> -- >> >>>>> "I think James Connolly, the Irish revolutionary, is right when he >> >>>>> says that the only prophets are those who make their future. So we're >> >>>>> not anticipating, we're working for it." >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> David Ragazzi >> >>>> OS Community Commiter >> >>>> Numenta.org >> >>>> -- >> >>>> "I think James Connolly, the Irish revolutionary, is right when he says >> >>>> that >> >>>> the only prophets are those who make their future. So we're not >> >>>> anticipating, we're working for it." >> >>>> >> >>>> _______________________________________________ >> >>>> nupic mailing list >> >>>> [email protected] >> >>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>>> >> >>>> >> >>> >> >>> _______________________________________________ >> >>> nupic mailing list >> >>> [email protected] >> >>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >>> >> >>> >> >> >> >> >> >> -- >> >> David Ragazzi >> >> OS Community Commiter >> >> Numenta.org >> >> -- >> >> "I think James Connolly, the Irish revolutionary, is right when he says >> >> that >> >> the only prophets are those who make their future. So we're not >> >> anticipating, we're working for it." >> >> >> >> _______________________________________________ >> >> nupic mailing list >> >> [email protected] >> >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> >> >> >> > >> > _______________________________________________ >> > nupic mailing list >> > [email protected] >> > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> > >> > >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
