[nox-dev] NOX packaging issues
Hi,
I am currently trying to create a Debian package for NOX. I encountered
many issues with NOX's autotools-based packaging that make it very
difficult to package, listed below.
I propose two strategies to handle those issues and obtain a NOX version
that can be properly made into a Debian package:
1) I have patches that solve almost all those issues. It's a lot of
work, it's not finished, and I'm not yet satisfied with the result. Are
you OK to do such an overhaul of NOX's build system? I.e. would you be
willing to review and integrate such patches?
2) It's such a frustrating work that I'd just prefer to fork NOX and do
a clean, minimal version including only a working nox-core
implementation and a new set of configure.ac / Makefile.am files redone
from scratch. The rest of the code could be packaged separately, from
another source.
(I think the name Forkt NOX would be good for this ;))
Which strategy do you think is better?
My issues:
- the use of a heavy-handed boot.sh script to configure the parts of the
system to build, instead of using Autoconf's standard AC_ARG_ENABLE
macros to add --enable-X / --disable-X options to the configure script,
and then defining and using variables that can be tested in Makefiles to
enable options;
- the use of a custom M4 macro for testing for Python, instead of using
Automake's AM_PATH_PYTHON which at least correctly selects the correct
Python version when multiple versions of Python are installed in the
system and defines correctly the installation directory for Python files
($pythondir):
- pythondir is calculated by AM_PATH_PYTHON using this code:
from distutils import sysconfig
print sysconfig.get_python_lib(0,0,prefix='$prefix')
i.e., this code correctly takes into account the prefix
- PYTHON_SITE_PKG is calcualted by AC_PYTHON_DEVEL (used by NOX)
using this code;
import distutils.sysconfig
print distutils.sysconfig.get_python_lib(0,0)
i.e., it doesn't take into account the configured prefix so
PYTHON_SITE_PKG cannot be used as-is as an installation dir
- at install time, all Python sources are installed in prefix/..., then
compiled using the configured version of Python, and then deleted for no
good reason; this makes it impossible to install NOX to support multiple
versions of Python in the same system. The Debian way, e.g. using
python-support, is to: first install the Python sources in a Python
version-independent directory, normally $prefix/share/pyshared/, and
then let python-support compile the sources for every version of Python
installed in the system, e.g. into
$prefix/share/python-2.6/site-packages/,
$prefix/share/python-2.4/site-packages/, etc.
- all the Python sources are installed into $prefix/bin, even Python
files that are not meant to be executables; this is just wrong: Python
sources should normally be installed into $prefix/share/python-*/ (or
preferably into $prefix/share/pyshared/), Python extension modules (.so
libs, etc.) should be installed into the dir for each supported version
of Python ($exec_prefix/lib/python-*/site-packages);
- just anything is installed along with the Python source files (listed
in the NOX_RUNTIMEFILES variables): I've found Swig *.i files, C++ *.cc
source code files, *.sh test scripts, config files (meta.xml) etc. Those
should really be installed separately, into different directories.
- specifically for the meta.xml, it would be better to install them into
a specific directory containing only those files, e.g. $pkgdatadir/
(i.e. $prefix/share/nox/...);
- test files should not be installed at all (cf. the .../t/ directories
for instance);
- the SSL certificates are generated at build time, included into the
dist archive, and then copied at install time. Without modification, it
would mean that all users installing from a dist archive or from a
Debian package made from this unmodified build system would use the same
SSL certificate, which of course must be absolutely avoided. The SSL
certificates should instead be generated much later, at package
installation time. They should at least not be generated at build time,
and not be included into the dist archive.
- there are several hard-coded paths in the source code, e.g. in
switch_command.py there is this line:
sys.path.append('/opt/nox/bin')
- a lot of files are just not listed in the Makefiles, and as a
consequence are not included in the dist archive (generated with make
dist), and sometimes not even built or installed. For instance, the
src/scripts is not even listed in the SUBDIRS of src/Makefile.am, so its
files cannot be included into the dist archive. Or the *.txt files in
src/tests/.
- some temporary files are not cleaned properly. For instance,
src/lib/dhparams.c should be in the CLEANFILES and should not even be
present in the Git repository;
- some Makefiles handle files in subdirs, e.g. src/include/Makefile.am
takes care of all the files in
Re: [nox-dev] NOX packaging issues
These changes are badly needed. The current build system is a long running amalgam of poor implementation decisions and a failed experiment to dynamically consume components as they are added. Having separate packages for noxcore (which may or may not include coreapps), netapps, and webapps is probably the right way to go about this. I know KK and Murphy have been thinking about distribution a bit, and should probably chime in. I definitely think we should just do a wholesale upgrade. The most current work on Nox is going on in the destiny branch. KK/Murphy, I know there has been some work in overhauling building/config, where does that stand? .. and ForkdNox is a great name. ;) .martin Hi, I am currently trying to create a Debian package for NOX. I encountered many issues with NOX's autotools-based packaging that make it very difficult to package, listed below. I propose two strategies to handle those issues and obtain a NOX version that can be properly made into a Debian package: 1) I have patches that solve almost all those issues. It's a lot of work, it's not finished, and I'm not yet satisfied with the result. Are you OK to do such an overhaul of NOX's build system? I.e. would you be willing to review and integrate such patches? 2) It's such a frustrating work that I'd just prefer to fork NOX and do a clean, minimal version including only a working nox-core implementation and a new set of configure.ac / Makefile.am files redone from scratch. The rest of the code could be packaged separately, from another source. (I think the name Forkt NOX would be good for this ;)) Which strategy do you think is better? My issues: - the use of a heavy-handed boot.sh script to configure the parts of the system to build, instead of using Autoconf's standard AC_ARG_ENABLE macros to add --enable-X / --disable-X options to the configure script, and then defining and using variables that can be tested in Makefiles to enable options; - the use of a custom M4 macro for testing for Python, instead of using Automake's AM_PATH_PYTHON which at least correctly selects the correct Python version when multiple versions of Python are installed in the system and defines correctly the installation directory for Python files ($pythondir): - pythondir is calculated by AM_PATH_PYTHON using this code: from distutils import sysconfig print sysconfig.get_python_lib(0,0,prefix='$prefix') i.e., this code correctly takes into account the prefix - PYTHON_SITE_PKG is calcualted by AC_PYTHON_DEVEL (used by NOX) using this code; import distutils.sysconfig print distutils.sysconfig.get_python_lib(0,0) i.e., it doesn't take into account the configured prefix so PYTHON_SITE_PKG cannot be used as-is as an installation dir - at install time, all Python sources are installed in prefix/..., then compiled using the configured version of Python, and then deleted for no good reason; this makes it impossible to install NOX to support multiple versions of Python in the same system. The Debian way, e.g. using python-support, is to: first install the Python sources in a Python version-independent directory, normally $prefix/share/pyshared/, and then let python-support compile the sources for every version of Python installed in the system, e.g. into $prefix/share/python-2.6/site-packages/, $prefix/share/python-2.4/site-packages/, etc. - all the Python sources are installed into $prefix/bin, even Python files that are not meant to be executables; this is just wrong: Python sources should normally be installed into $prefix/share/python-*/ (or preferably into $prefix/share/pyshared/), Python extension modules (.so libs, etc.) should be installed into the dir for each supported version of Python ($exec_prefix/lib/python-*/site-packages); - just anything is installed along with the Python source files (listed in the NOX_RUNTIMEFILES variables): I've found Swig *.i files, C++ *.cc source code files, *.sh test scripts, config files (meta.xml) etc. Those should really be installed separately, into different directories. - specifically for the meta.xml, it would be better to install them into a specific directory containing only those files, e.g. $pkgdatadir/ (i.e. $prefix/share/nox/...); - test files should not be installed at all (cf. the .../t/ directories for instance); - the SSL certificates are generated at build time, included into the dist archive, and then copied at install time. Without modification, it would mean that all users installing from a dist archive or from a Debian package made from this unmodified build system would use the same SSL certificate, which of course must be absolutely avoided. The SSL certificates should instead be generated much later, at package installation time. They should at least not be generated at build time, and not be included into the dist archive. - there are several hard-coded paths in the source code, e.g. in
Re: [nox-dev] NOX packaging issues
It mostly stands as it doesn't seem worth it to get into it. We were interested in improving modularity some (and the idea of replacing the build system wholesale did come up), but it became clear that it was going to be a lot of effort to do something that wasn't necessary to achieve our more immediate goal for anyway. That goal, BTW, is to move the standard for third-party component development from building them inside the tree to building them outside the tree. I think this simplifies things somewhat, but another (open) question is: are we willing to throw out the idea of doing static builds? If we're not worried about that, I think that's another simplification. Some thoughts: I'm worried about making changes that are going to be Debian-specific (e.g., relying on update-python-modules). I'm not sure I agree re: installing meta.xml/meta.json to a centralized location. This makes sense for a world in which the norm is for people to install a NOX package and a bunch of component packages, maybe. But I think it makes less sense for people doing research and experiments with NOX. I think encapsulating a component in a directory works pretty well. At the minimum, it seems like we'd need a way to specify additional directories of metafiles on the commandline or something like that. Besides that, this sounds really good, and I think we could put a lot of these improvements into the destiny branch (by the way, make dist in destiny is somewhat improved over master already in that it actually does at least sort of work). -- Murphy On Aug 6, 2010, at 6:57 AM, Martin Casado wrote: These changes are badly needed. The current build system is a long running amalgam of poor implementation decisions and a failed experiment to dynamically consume components as they are added. Having separate packages for noxcore (which may or may not include coreapps), netapps, and webapps is probably the right way to go about this. I know KK and Murphy have been thinking about distribution a bit, and should probably chime in. I definitely think we should just do a wholesale upgrade. The most current work on Nox is going on in the destiny branch. KK/Murphy, I know there has been some work in overhauling building/config, where does that stand? .. and ForkdNox is a great name. ;) .martin Hi, I am currently trying to create a Debian package for NOX. I encountered many issues with NOX's autotools-based packaging that make it very difficult to package, listed below. I propose two strategies to handle those issues and obtain a NOX version that can be properly made into a Debian package: 1) I have patches that solve almost all those issues. It's a lot of work, it's not finished, and I'm not yet satisfied with the result. Are you OK to do such an overhaul of NOX's build system? I.e. would you be willing to review and integrate such patches? 2) It's such a frustrating work that I'd just prefer to fork NOX and do a clean, minimal version including only a working nox-core implementation and a new set of configure.ac / Makefile.am files redone from scratch. The rest of the code could be packaged separately, from another source. (I think the name Forkt NOX would be good for this ;)) Which strategy do you think is better? My issues: - the use of a heavy-handed boot.sh script to configure the parts of the system to build, instead of using Autoconf's standard AC_ARG_ENABLE macros to add --enable-X / --disable-X options to the configure script, and then defining and using variables that can be tested in Makefiles to enable options; - the use of a custom M4 macro for testing for Python, instead of using Automake's AM_PATH_PYTHON which at least correctly selects the correct Python version when multiple versions of Python are installed in the system and defines correctly the installation directory for Python files ($pythondir): - pythondir is calculated by AM_PATH_PYTHON using this code: from distutils import sysconfig print sysconfig.get_python_lib(0,0,prefix='$prefix') i.e., this code correctly takes into account the prefix - PYTHON_SITE_PKG is calcualted by AC_PYTHON_DEVEL (used by NOX) using this code; import distutils.sysconfig print distutils.sysconfig.get_python_lib(0,0) i.e., it doesn't take into account the configured prefix so PYTHON_SITE_PKG cannot be used as-is as an installation dir - at install time, all Python sources are installed in prefix/..., then compiled using the configured version of Python, and then deleted for no good reason; this makes it impossible to install NOX to support multiple versions of Python in the same system. The Debian way, e.g. using python-support, is to: first install the Python sources in a Python version-independent directory, normally $prefix/share/pyshared/, and then let python-support compile the sources for every version of Python installed
Re: [nox-dev] attribute '_handleSigchld'
Hi Tim! You are correct. After adding the following function definition to '/usr/lib/python2.6/dist-packages/twisted/internet/base.py', the problem seems to be gone. def _handleSigchld(self, signum, frame, _threadSupport=platform.supportsThreads()): from twisted.internet.process import reapAllProcesses if _threadSupport: self.callFromThread(reapAllProcesses) else: self.callLater(0, reapAllProcesses) Thanks a bunch! Vjeko On Fri, Aug 06, 2010 at 04:30:28PM -0400, tim.upthegr...@gatech.edu wrote: Hi Vjekoslav, I had this issue before after doing a fresh install on a Debian sid machine using the openflow-1.0 branch. What I found is, the function _handleSigchld has been moved-around/removed from more recent versions of TwistedPython. The fix I found was to go grab Twisted 8.0 and install it. I haven't tested the controller in a production setup yet, but the source compiles and I can run a simple switch without getting any error messages. Give that a shot and see if it works. Best of luck, Tim Upthegrove Georgia Institute of Technology College of Computing (406)-437-1493 - Vjekoslav Brajkovic bal...@cs.washington.edu wrote: Hello. After I upgraded my system, I've been having trouble starting NOX. This is the error message: 2|nox|ERR:Cannot change the state of 'python' to INSTALLED: 'python' ran into an error: Unable to construct a Python component: Traceback (most recent call last): File ./nox/coreapps/pyrt/pyoxidereactor.py, line 364, in instance return pyoxidereactor(ctxt) File ./nox/coreapps/pyrt/pyoxidereactor.py, line 133, in __init__ signal.signal(signal.SIGCHLD, self._handleSigchld) AttributeError: 'pyoxidereactor' object has no attribute '_handleSigchld' Any clues to why this might be happening? Thanks! Vjeko ___ nox-dev mailing list nox-dev@noxrepo.org http://noxrepo.org/mailman/listinfo/nox-dev_noxrepo.org ___ nox-dev mailing list nox-dev@noxrepo.org http://noxrepo.org/mailman/listinfo/nox-dev_noxrepo.org
Re: [nox-dev] NOX packaging issues
Hi, While I am a huge fan of apt-get, I do have some reservation about NOX package at this moment. Researchers want to work with source code. I am not aware of people who want NOX binaries for installation, probably with the exception of Guido. :P Nonetheless, cleaning up the build system would be much appreciated. I would suggest working on the destiny branch where a significant number of changes have been made. I am personally always happy to get more help. This can be one-shot help to clean the build system too. Whether Romain prefers. My preference now is to get people to be able to build out source, so that they can distribute their NOX components independently, and use other third party components. If we can have a source package for NOX that will install in user directory and all the dependencies, that would be great too. I am still interested to hear if anyone wants a NOX binary. This is a good goal to have in the long term regardless of what I have said above. If I can have it now with the above, I will take it. Regards KK On 6 August 2010 10:17, James Murphy McCauley jam...@nau.edu wrote: It mostly stands as it doesn't seem worth it to get into it. We were interested in improving modularity some (and the idea of replacing the build system wholesale did come up), but it became clear that it was going to be a lot of effort to do something that wasn't necessary to achieve our more immediate goal for anyway. That goal, BTW, is to move the standard for third-party component development from building them inside the tree to building them outside the tree. I think this simplifies things somewhat, but another (open) question is: are we willing to throw out the idea of doing static builds? If we're not worried about that, I think that's another simplification. Some thoughts: I'm worried about making changes that are going to be Debian-specific (e.g., relying on update-python-modules). I'm not sure I agree re: installing meta.xml/meta.json to a centralized location. This makes sense for a world in which the norm is for people to install a NOX package and a bunch of component packages, maybe. But I think it makes less sense for people doing research and experiments with NOX. I think encapsulating a component in a directory works pretty well. At the minimum, it seems like we'd need a way to specify additional directories of metafiles on the commandline or something like that. Besides that, this sounds really good, and I think we could put a lot of these improvements into the destiny branch (by the way, make dist in destiny is somewhat improved over master already in that it actually does at least sort of work). -- Murphy On Aug 6, 2010, at 6:57 AM, Martin Casado wrote: These changes are badly needed. The current build system is a long running amalgam of poor implementation decisions and a failed experiment to dynamically consume components as they are added. Having separate packages for noxcore (which may or may not include coreapps), netapps, and webapps is probably the right way to go about this. I know KK and Murphy have been thinking about distribution a bit, and should probably chime in. I definitely think we should just do a wholesale upgrade. The most current work on Nox is going on in the destiny branch. KK/Murphy, I know there has been some work in overhauling building/config, where does that stand? .. and ForkdNox is a great name. ;) .martin Hi, I am currently trying to create a Debian package for NOX. I encountered many issues with NOX's autotools-based packaging that make it very difficult to package, listed below. I propose two strategies to handle those issues and obtain a NOX version that can be properly made into a Debian package: 1) I have patches that solve almost all those issues. It's a lot of work, it's not finished, and I'm not yet satisfied with the result. Are you OK to do such an overhaul of NOX's build system? I.e. would you be willing to review and integrate such patches? 2) It's such a frustrating work that I'd just prefer to fork NOX and do a clean, minimal version including only a working nox-core implementation and a new set of configure.ac / Makefile.am files redone from scratch. The rest of the code could be packaged separately, from another source. (I think the name Forkt NOX would be good for this ;)) Which strategy do you think is better? My issues: - the use of a heavy-handed boot.sh script to configure the parts of the system to build, instead of using Autoconf's standard AC_ARG_ENABLE macros to add --enable-X / --disable-X options to the configure script, and then defining and using variables that can be tested in Makefiles to enable options; - the use of a custom M4 macro for testing for Python, instead of using Automake's AM_PATH_PYTHON which at least correctly selects the correct Python version when multiple
Re: [nox-dev] NOX packaging issues
Hi James, Thanks for your comments. My replies are inline below. On 2010-08-07 02:17, James Murphy McCauley wrote: It mostly stands as it doesn't seem worth it to get into it. We were interested in improving modularity some (and the idea of replacing the build system wholesale did come up), but it became clear that it was going to be a lot of effort to do something that wasn't necessary to achieve our more immediate goal for anyway. That goal, BTW, is to move the standard for third-party component development from building them inside the tree to building them outside the tree. I think this simplifies things somewhat, but another (open) question is: are we willing to throw out the idea of doing static builds? If we're not worried about that, I think that's another simplification. Some thoughts: I'm worried about making changes that are going to be Debian-specific (e.g., relying on update-python-modules). I will not make any Debian-specific changes. Using Debian's dh_pysupport, make install only needs to copy the Python source files into /usr/share/python-*/site-packages/ (the standard location), and dh_pysupport will copy those files into /usr/share/pyshared/site-packages/ at package generation time and do all the Debian-specific stuff. But right now, not installing the Python sources and keeping only compiled Python modules in a non-standard location doesn't work for Debian packaging. I'm not sure I agree re: installing meta.xml/meta.json to a centralized location. This makes sense for a world in which the norm is for people to install a NOX package and a bunch of component packages, maybe. But I think it makes less sense for people doing research and experiments with NOX. I think encapsulating a component in a directory works pretty well. At the minimum, it seems like we'd need a way to specify additional directories of metafiles on the commandline or something like that. The problem with the current system is that NOX requires files to be named only 'meta.xml', i.e. if one wants to have two components in the same Python package, this requires to specify them in the same meta.xml file which prevents packaging the two components separately, or to create subdirectories just to be able to have multiple meta.xml which is cumbersome. I was thinking of something like the /etc/*.d/ directories (e.g. /etc/cron.d/). More specifically I propose to: - accept any files named /usr/share/nox/**/*.xml, instead of requiring them to be called meta.xml; - look for files in additional directories by default, e.g. /usr/local/share/nox/**/*.xml. Since the component namespace is flat, a good way to handle collisions would be to recommend users to name their *.xml files after the names of the components they specify: switch.xml, topology.xml, etc. Besides that, this sounds really good, and I think we could put a lot of these improvements into the destiny branch (by the way, make dist in destiny is somewhat improved over master already in that it actually does at least sort of work). OK, I'll take a look at the destiny branch. Best regards, -- Romain Lenglet ___ nox-dev mailing list nox-dev@noxrepo.org http://noxrepo.org/mailman/listinfo/nox-dev_noxrepo.org
Re: [nox-dev] NOX packaging issues
Hi KK, On 2010-08-07 08:28, kk yap wrote: Hi, While I am a huge fan of apt-get, I do have some reservation about NOX package at this moment. Researchers want to work with source code. I am not aware of people who want NOX binaries for installation, probably with the exception of Guido. :P Nonetheless, cleaning up the build system would be much appreciated. I would suggest working on the destiny branch where a significant number of changes have been made. I am personally always happy to get more help. This can be one-shot help to clean the build system too. Whether Romain prefers. I'll take a look at the destiny branch, and see what's left to be done. My preference now is to get people to be able to build out source, so that they can distribute their NOX components independently, and use other third party components. If we can have a source package for NOX that will install in user directory and all the dependencies, that would be great too. What I want to do is to make the build system more standard, i.e. install files in standard locations, etc. That's necessary for Debian packaging, but it will also certainly help users that manage local installs in /usr/local/ or in their homedirs, e.g. using GNU Stow. I am still interested to hear if anyone wants a NOX binary. This is a good goal to have in the long term regardless of what I have said above. If I can have it now with the above, I will take it. Regards KK Thanks for your comments, Best regards, -- Romain Lenglet ___ nox-dev mailing list nox-dev@noxrepo.org http://noxrepo.org/mailman/listinfo/nox-dev_noxrepo.org
