Hi, I had a look at the code, please find my remarks for each point in the text bellow:
On Wed, 2007-11-07 at 00:04 +0100, Erich Focht wrote: > Hi, > > here is an attempt to document the changes made for getting a 5.1 alike > branch to work. Besides the opkgc bugs (thanks to DongInn for helping > to fix them). > > > 1) Database schema change > > Eliminated package_id as reference from several tables to packages. Replaced > by package (name), a string, but as good as an integer index. > This has the advantage of making sense even when the package record is changed > and the id changes, e.g. when the package version changes. > Replaced group_id by group_name in some places. > Added "distro" field to the package record, this is for multi-distro support. > The Database.pm interfaces changed minimally, the content of the functions was > simplified. Database.pm shrinked by 200 lines of code but is still to big. As i said during the call, what you did will simplify a lot of things for the management of partitions, point i started to work on. It breaks PSM and my version of OPM but i will deal with that since your modification improves things. BTW, since you did these modifications, maybe we can think about the partitioning issue and make sure that the database is ready to store such information? Doing so we will not have to modify again the database schema in few months. What do you think about that? > > 2) OSCAR::OpkgDB > > This is in trunk. OpkgDB are database-like functions querying the remote > YUM / APT repositories and/or the local RPM/DEB database instead of ODA. > The functions return info about opkg-* metapackages (api, server, client) > and return similar values like the database queries (arrays of hashes with > the field keys as index). They are designed to replace ODA queries (which > address cached package info) by direct queries of the package manager or > repository databases. A performance problem might exist but is normally > uncritical. Do we agree that mid-term we have to modify Packman to provide a simple interface to query repositories? About your remark on ODA and the performance issue, i do not agree with that and it may lead to issues in specific cases but we can discuss about that during SC. A example: you have an image, deployed node based on this image; you want to install a new software on compute nodes. For that you have different approaches: - you can install OPKGs directly on compute nodes without to have to install them in the image. In that case, if you want to get the configuration of compute nodes, you have to query the compute nodes. If an application is running this is definitively not acceptable and there is no way to argue against that (this is one of our strong constraints: do not pollute the application execution). - you install the OPKG in the image and then "rsync" the compute nodes: it works, no need to have information into the database, if we need to know the configuration of the compute nodes, we just query the image configuration. Note that most of the time, users do not think that way so if we go in that direction we will have to make this kind of stuff simple to do and well integrated. - you enforce users to install the OPKGs in the image and on compute nodes; and therefore you do not need information from the database, we only have to query the image in order to get information about the current configuration. This is definitively not natural for users, we have a lot of tools to achieve that. This is the kind of stuff for which i would like to have a clear idea of what you plan to do before to remove the "double caching mechanism". BTW, in my mind, the double caching is only optimization: every time you want to interact (install/remove/update) with OPKG we use a tool similar to APT and so on: before to do something, on the headnode only, you check if information you have are up-to-date; if no, you notify the user; if yes, you do what you have to do, no problem. I do not see why this approach is bad. Also remember that up to now we assumed that the database was the central piece that abstract all information. It is for instance possible to imagine to have a remote management GUI that just need to connect to the database and execute few commands remotely. If you by-pass the database, this kind of stuff will be much more difficult to implement (i think) and it is a clear paradigm shift in the OSCAR architecture. Again i am open to discussions on this point; it is just that today, i do not understand why it is so critical to by-pass ODA for this kind of stuff. In my mind we still need ODA. > 3) OSCAR::PackagePath > > Added functionality to manage the repositories for a distro. They are stored > in the files /tftpboot/oscar/<compat_distro>-<version>-<arch>.url . This was > anyway the place where the repsitories could be configured. yume received > an interface for controlling these repositories from the command line: > --repoadd|--repodel|--repolist . But it is only able to modify the > repositories > of the locally detected distro. For other distros: use PackagePath. > By the way: list_distro_pools can be used to enumerate the configured distros, > i.e. those for which distro (not oscar) repositories are configured. I started to use that for the new OPD in order to manage repos. :-) very useful. > > 4) OSCAR::Opkg > > Some extensions for opkg handling and installation. Main difference to > trunk: replaced opkg_install_server() by a more general opkg_install(). Good. > > 5) scripts/populate_oda_packages_table > > Used for compatibility reasons. Populates the new Packages table with info > retrieved from repositories about the opkg-* metapackages. Can accept as > argument a string describing the distro, in which case it will query the > corresponding distro and feed info on those packages to ODA. Very good. > > > 6) wizard_prep logic change > > - prereqs install > - install all available opkg-API metapackages (for the master node) on > the master node. This brings the opkg API scripts to the master node, > including the setup scripts. Before: API scripts came with OSCAR. Now they > come from the repository during the startup of the wizard. If new repositories > were added, restart the wizard. > - run package setup scripts (like before) > - create a hash array of the locally installed opkgs (needed for install > steps). > Is equivalent to a database query, but until now we actually would not need > the > database to be up. Soon that will not make much sense i think. This is only due to the fact that we assume the current node is the headnode and the single server in the plaftorm. > - prepare_oda (starts mysql, creates the tables?) > - populate_default_package_set (not sure this script works, actually) > - populate the default package set from the share/package_sets/Default/*.xml > (this needs XML::Simple, but actually what we manage (a list of opkgs) > isn't really worth a xml file and all the infrastructure around it. I do not know why you tried to use PSM for that, as i said during a call, PSM is NOT stable and should NOT be used currently. The PackageSet.pm module does everything you want to do. For instance the functions get_local_package_set_list and get_list_opkgs_in_package_set should allow you to do what you want to implement. No need to rewrite code from scratch, to duplicate code. > - create default package configs by interpreting their configurator.html > files. BTW, note that for what i remember, the parsing of HTML file with a CLI is not trivial, we should address this issue later (not now, it is only a side note). > - set_node_nics > - install core opkg-*-server packages > - mark packages installed in Node_Package_Status table > - run post_server_install scripts for the core packages > - export local pools to clients through http. > > 7) Fixed Selector to manage package sets > > Re-enabled package sets management. Use OpkgDB queries instead of ODA for > populating tables. Sets are created in the Group_Packages table and package > group > files are written in $OSCAR_HOME/tmp/*.pgroup > All set management functions work: duplicate, rename, create all/core, delete. > Fixes from DongInn for package manipulation are included. We clearly duplicate efforts here. I do not know what to say because everybody clearly know we, at ORNL, are currently working on that and nobody did even discuss with us. :-( I guess I have to read more carefully the code and try to figure out what we can do. > > 8) Added functionality to SystemInstaller for selecting package groups when > building images. These are passed from the outside, if nothing is passed, > things work like before. Same remark. > > The package groups functionality is problematic because the master gets the > _selected_ package set from Selector, i.e. the one which is active when > closing > the Selector. If this set is not the union of the other sets, the master node > might miss packages. Yes, this is a so simple point and that's why we are working on that. > This needs to be fixed, a superset of all packages can be > created easilly on the fly. For now: make sure you select a big enough set on > the master node. Oh, there is a limitation: we cannot handle requirements and > conflicts of opkgs right now. Also the servicelists and switcher info is > missing. The reason is: opkgc cannot put this into rpms right now. We need to > find a way to transport this information. An example: we multiplex class and > group information into the Group: field of RPMs. Need something similar for > DEBs, and for the additional info. > > There might be more changes, but I don't remember. I also updated some > packages > (e.g. systemimager-4.0.0) but these are merged to trunk. I am happy with most of the changes but clearly some decisions seem to be in conflict with what we are doing/did. Everybody knew we were working on these subjects and it seems that decisions were made for the branch without discussions with us. I do not know what to say right now, i have to read the code in more details and see if i can come with a solution. > > Regards, > Erich > > ------------------------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. > Still grepping through log files to find problems? Stop. > Now Search log events and configuration files using AJAX and a browser. > Download your FREE copy of Splunk now >> http://get.splunk.com/ > _______________________________________________ > Oscar-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/oscar-devel ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Oscar-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/oscar-devel
