Re: [GSoC] migration
> On Jul 19, 2017, at 3:43 PM, Umesh Singla wrote: > > Hi Brad, > > With another week over, I have the following updates to share. > > > GSoC Week #7 (13 Jul - 19 Jul) > > I moved the things to migrate action instead of 'restore' since 'restore' > looks a subset of 'migrate', so can be dealt easily once migrate is over. I’m not sure I understand and if I do understand I think I disagree. Snapshot is a state of installed and active ports with their variants. Restoring a snapshot would turn off all parts and install/activate the snapshot ports with variants. Migration would be taking a new snapshot, uninstalling all ports, upgrading the port command for the new arch and finally restoring the last snapshot which would entail installing all ports from snapshot since we installed earlier. Snapshot and restore should be able to be executed without migrate. Reading below I think I misunderstand so please clarify. > Agenda for last week: > > The migrate action consisted of three parts, creating a new snapshot, > uninstall installed ports, re-installing them for the last snapshot. > > 1. Creating a new snapshot was over before, though some refactoring is still > left. > > 2. Regarding uninstalling installed ports, I took the help of reclaim.tcl > which works with the migrate action now. It uses > sort_portlist_by_dependendents procedure. So, all in all, it uses > "registry_uninstall::uninstall". > > 3. Re-installing the ports from the portlist. I took the help of > restore_ports.tcl here, which works when called migrate. It uses sort_ports > [1] function. > > > > When we install, is it fine to use [mportexec $workername $install_target] > from migrate.tcl? I’m sorry, I don’t know base well enough to answer this. Can you please ask Clemens or Josh? > What took time is that these two functions at present take the port list in > different formats. So, I tried debugging by simply printing on console method > to know the formats of the two. > > I want to use the sorting function for dependencies only once instead of > calling it twice while uninstalling and then re-installing. I'm still working > on it and actually, close to solve this but thought, better send the weekly > update first. It is indeed better to send updates more frequently. I’d check with Clemens on this. He mentored a previous GSoC project which used libsolve for dependency calculations. Perhaps there is something to borrow or learn from that project? > For the next one week: > > • make appropriate changes to procedures according to one format. > • write for “SELECT PORTS FROM SNAPSHOT ..”, i.e. connect it with the > new tables we have. > • try finishing the migrate action asap? Sounds good. > Points: > > Since migrate action creates a snapshot within and then uses it to redo the > whole state, I feel creating a snapshot here explicitly is redundant. Not sure I follow. I imaging migrate action would start by calling snapshot, register the snapshot id, uninstall all ports, reinstall the port command and finally call restore action with the snapshot id for the last snapshot. > Internally snapshot also uses the same registry C functions to get the > information first and then save to database as [registry::entry imaged] does, > so instead of first retrieving from [registry::entry imaged], saving it in > sqlite and then again fetching all the info using the snapshot-id we just > got, can't we directly use [registry::entry imaged] here? Am I missing > something? Oh, maybe in this case we already have the list in memory? But when we upgrade the port command wouldn’t we then call the new port command and perform the restore with the upgrade port command? > Regarding "selected" snapshot for 'restore', it looks fine. > > New points which I made during pondering to save for the end: > > • Add ui_debug statements > • Catch some error-prone areas. > • Remove the useless/structure comments and add actual doc-strings. > • Add copyright notices > • Using ui_msg instead of puts and else. Should probably add that. Yes to all the above. > Also, here's my view of restore_ports.tcl of what it does: > > • macports prefix configs > • umask 022 > • set portList [read_portlist $filename] - reads the portlist from file > supplied or stdin > • set operationList [sort_ports $portList] > • sort_ports $portList > • extracts name, version, variants, active for a port > • sets it 1 in port_in_list if not there, else > increments the count for that port > • if port dependencies for the combination (port_name, > its variants) doesn’t exist already, fetches them using > dependenciesForPort(port_name, its variants), kind of recursively. > • dependenciesForPort $port_name $variants > • install_ports $operationList > • checks $active
Re: [GSoC] migration
Hi Brad, With another week over, I have the following updates to share. *GSoC Week #7 (13 Jul - 19 Jul)* I moved the things to migrate action instead of 'restore' since 'restore' looks a subset of 'migrate', so can be dealt easily once migrate is over. Agenda for last week: The migrate action consisted of three parts, creating a new snapshot, uninstall installed ports, re-installing them for the last snapshot. 1. Creating a new snapshot was over before, though some refactoring is still left. 2. Regarding uninstalling installed ports, I took the help of reclaim.tcl which works with the migrate action now. It uses sort_portlist_by_dependendents procedure. So, all in all, it uses "registry_uninstall::uninstall". 3. Re-installing the ports from the portlist. I took the help of restore_ports.tcl here, which works when called migrate. It uses sort_ports [1] function. When we install, is it fine to use [mportexec $workername $install_target] from migrate.tcl? What took time is that these two functions at present take the port list in different formats. So, I tried debugging by simply printing on console method to know the formats of the two. I want to use the sorting function for dependencies only once instead of calling it twice while uninstalling and then re-installing. I'm still working on it and actually, close to solve this but thought, better send the weekly update first. For the next one week: 1. make appropriate changes to procedures according to one format. 2. write for “SELECT PORTS FROM SNAPSHOT ..”, i.e. connect it with the new tables we have. 3. try finishing the migrate action asap? Points: Since migrate action creates a snapshot within and then uses it to redo the whole state, I feel creating a snapshot here explicitly is redundant. Internally snapshot also uses the same registry C functions to get the information first and then save to database as [registry::entry imaged] does, so instead of first retrieving from [registry::entry imaged], saving it in sqlite and then again fetching all the info using the snapshot-id we just got, can't we directly use [registry::entry imaged] here? Am I missing something? Regarding "selected" snapshot for 'restore', it looks fine. New points which I made during pondering to save for the end: 1. Add ui_debug statements 2. Catch some error-prone areas. 3. Remove the useless/structure comments and add actual doc-strings. 4. Add copyright notices 5. Using ui_msg instead of puts and else. Should probably add that. Also, here's my view of restore_ports.tcl of what it does: - macports prefix configs - umask 022 - set portList [read_portlist $filename] - reads the portlist from file supplied or stdin - set operationList [sort_ports $portList] - sort_ports $portList - extracts name, version, variants, active for a port - sets it 1 in port_in_list if not there, else increments the count for that port - if port dependencies for the combination (port_name, its variants) doesn’t exist already, fetches them using dependenciesForPort(port_name, its variants), kind of recursively. - dependenciesForPort $port_name $variants - install_ports $operationList - checks $active for a port and use it to decide install_target - uses mportexec for installing Please find the work log here [2]. It is largely in work-in-progress state. [1]: https://github.com/macports/macports-contrib/blob/master/restore_ports/restore_ports.tcl#L52 [2]: https://github.com/macports/macports-base/commits/gsoc17-migrate Best Regards, Umesh Singla
combining directories / moving files within Portfile
I have an issue with SIP that needs to be addressed, that I'm not sure how do execute in port via Portfiles. I have a solution, but it's kinda a hack. Background --- Some SIP dependencies use sipconfig to glean the directory into which to install generated SIP files [in Python: "import sipconfig"; "sipcfg = sipconfig.Configuration()"; "sipcfg.default_sip_dir" -> default "--sipdir" for installing generated SIP files]. Other SIP dependencies use a configuration --sipdir for this directory, with the default generally being "$prefix/share/pyXY-sip/$subport", but sometimes "$PYTHON_PREFIX/share/sip/$subport" where "PYTHON_PREFIX" is the top-level Python directory (in MacPorts, "$frameworks_dir/Python.framework/Versions/X.Y/"). I've been told that the "sipconfig" import is being removed from SIP sometime soonish, so we're better off specifying a directory into which ports install SIP files. Also, different ports choose different SIP directories, and combining these makes sense. What I'd like to do is symlink "$prefix/share/pyXY-sip/$subport" -> "$PYTHON_PREFIX/share/sip/$subport", so that it doesn't matter which method the port chooses, even if we specify one directory or the other. I can fix all of the ports that I know of that use SIP to install into the PYTHON_PREFIX version, but the ordering makes a difference because both of the SIP directories are populated at least on my MacPorts install. This poses the issue that the SIP port should be creating the latter directory as well as symlink, but it cannot while the former directory has anything in it. I can update the dependencies first, so that they are all installing into the PYTHON_PREFIX directory, then update SIP ... but, if someone does a "sudo port selfupdate" then port will do the update with SIP first followed by dependencies, which won't work. I can do this & update SIP maybe a year down the road ... but, that seems silly ... there has got to be a better way! I'd prefer to not require the user to deactivate various ports, instead automating the process of moving the files to allow the symlink to be created by the SIP port. My Hack --- My "hack" is to have SIP check in pre-activate for the directory "$prefix/share/pyXY-sip/" and if found then create an archive of its contents (into /tmp/something.tar), and "cd $prefix/share && rm -rf pyXY-sip". Then the 'activate' takes place, which creates the symlink to the "$PYTHON_PREFIX/share/sip/$subport" directory. Then in post-activate check for the archive & if found restore the contents into the directory "$PYTHON_PREFIX/share/sip/" && remove the archive. In my testing, this hack works perfectly & without user intervention. That said, it's a hack & maybe not the right thing to do. Hence me reaching out to MP devs: is there a good way to combine the contents of multiple directories via a common port? Should I just give up on trying to automate the process, instead force the user to deactivate / reactivate ports? Thanks for your thoughts / comments. - MLD
