Re: GSoC'17: Add migrate action to port command Project

[Bradley Giesbrecht]

> On Mar 30, 2017, at 12:13 PM, Clemens Lang  wrote:
> 
> Hey everybody,
> 
> On Tue, Mar 28, 2017 at 01:37:55PM -0700, Bradley Giesbrecht wrote:
>>> On Mar 28, 2017, at 11:55 AM, Umesh Singla  wrote:
>> 
>> Does the migrate and snapshot actions help you divide the work? The
>> snapshot action is basically an inventory of what is installed along
>> with meta data like requested and variants. We will store snapshots in
>> our sqlite database.
>> 
>> It is early on but I can imagine a schema like:
>> 
>> Tables
>> snapshots (
>>  id,
>>  datetime,
>>  note
>> )
>> snapshot_ports (
>>  id,
>>  snapshots_id,
>>  port_name,
>>  requested
>> )
>> snapshot_port_variants (
>>  id,
>>  snapshot_ports_id,
>>  variant_sign, // (we allow plus and minus variants like +variant1 
>> -variant2)
>>  variant_name
>> )
> 
> This follows all the good rules of data normalization, but MacPorts
> internally uses a different representation of variants. Not saying I
> prefer that, just saying that we already have code to deal with it.
> 
> I would also not mind changing the format. It's a detail we can ignore
> until the project starts, though.
> 
> I've also left this as a comment in the proposal.
> 
> 
>>> I don't know if we'll allow Tcl to run in the new environment.
>>> Otherwise, it seems to implement port with a bash script calling on
>>> sqlite3 on the macports registry/registry.d to get the relevant
>>> data. 
>> 
>> I think we can require any version of port that includes the migrate
>> action, I do not think we will need bash. Clemens, others, do you
>> concur?
> 
> I would suggest that the first action of a 'port snapshot'/'port
> migration' action would be to recompile base, i.e. the equivalent of
> 'port -f selfupdate'. This would allow us to implement this completely
> in Tcl/C and use the existing abstractions and helper functions that
> MacPorts already has.
> 
>>> Also, will the project phasing out dependency on XCode tools affect
>>> this one anytime soon?
> 
> I won't, but thanks for keeping it in mind.
> 
> I've also left some other comments in your proposal. Feel free to take
> a look.
> 
> 
> On Wed, Mar 29, 2017 at 01:38:23PM +0200, Mojca Miklavec wrote:
>> One of the biggest pains at the moment is the fact that we don't
>> record requested variants anywhere. We do record requested ports, but
>> not the variants.
> 
> I agree that's a pain point we should change, but I don't think that
> should be covered by this proposal, unless it turns out to be way too
> small.
> 
> 
>> I would like to see one more scenario covered if there is sufficient
>> time. When users change some setting in their macports configuration
>> (I have https://trac.macports.org/wiki/LibcxxOnOlderSystems in mind,
>> but there might be other cases), it would be super helpful if users
>> with a working MacPorts installation could run
>>sudo port 
>> which would:
>> - remember which packages are installed
>> - deactivate all of them (and perhaps move the files to some temporary
>> location just in case they are needed again)
>> - reinstall the same packages using the new settings
>> 
>> Basically the same scenario as with regular migration, except that the
>> user would not upgrade the OS, only some of the basic macports
>> settings (OK, I'm not sure how exactly you planned to implement the
>> migration tool, I'm just speculating).
> 
> Good idea, that could just be a run of 'port snapshot' and 'port
> migrate' without the base update part.

Or migrate could call a “restore” action:

port snapshot // create a snapshot
port restore // restore a snapshot, default to last snapshot
port migrate // port snapshot, port selfupdate and port restore

And restore can accept a snapshot id:
port restore snapshot=20170330154118


—
Brad

Re: GSoC'17: Add migrate action to port command Project

[Umesh Singla]

Hi cal, ijackson

Thanks for reviewing my proposal and providing a great feedback. I've
replied to most of your comments in the proposal. This surely helps. Please
let me know if I have made some false assumption about anything, the
timeline/ project plan needs to be modified a great bit or I haven't talked
about some issue (like hardware, performance, ) which you were expecting
etc. These things can affect my application, so I'm just asking for more
guidance.

Thank you so much,
Umesh

Re: GSoC'17: Add migrate action to port command Project

[Clemens Lang]

Hey everybody,

On Tue, Mar 28, 2017 at 01:37:55PM -0700, Bradley Giesbrecht wrote:
> > On Mar 28, 2017, at 11:55 AM, Umesh Singla  wrote:
> 
> Does the migrate and snapshot actions help you divide the work? The
> snapshot action is basically an inventory of what is installed along
> with meta data like requested and variants. We will store snapshots in
> our sqlite database.
> 
> It is early on but I can imagine a schema like:
> 
> Tables
> snapshots (
>   id,
>   datetime,
>   note
> )
> snapshot_ports (
>   id,
>   snapshots_id,
>   port_name,
>   requested
> )
> snapshot_port_variants (
>   id,
>   snapshot_ports_id,
>   variant_sign, // (we allow plus and minus variants like +variant1 
> -variant2)
>   variant_name
> )

This follows all the good rules of data normalization, but MacPorts
internally uses a different representation of variants. Not saying I
prefer that, just saying that we already have code to deal with it.

I would also not mind changing the format. It's a detail we can ignore
until the project starts, though.

I've also left this as a comment in the proposal.


> > I don't know if we'll allow Tcl to run in the new environment.
> > Otherwise, it seems to implement port with a bash script calling on
> > sqlite3 on the macports registry/registry.d to get the relevant
> > data. 
> 
> I think we can require any version of port that includes the migrate
> action, I do not think we will need bash. Clemens, others, do you
> concur?

I would suggest that the first action of a 'port snapshot'/'port
migration' action would be to recompile base, i.e. the equivalent of
'port -f selfupdate'. This would allow us to implement this completely
in Tcl/C and use the existing abstractions and helper functions that
MacPorts already has.

> > Also, will the project phasing out dependency on XCode tools affect
> > this one anytime soon?

I won't, but thanks for keeping it in mind.

I've also left some other comments in your proposal. Feel free to take
a look.


On Wed, Mar 29, 2017 at 01:38:23PM +0200, Mojca Miklavec wrote:
> One of the biggest pains at the moment is the fact that we don't
> record requested variants anywhere. We do record requested ports, but
> not the variants.

I agree that's a pain point we should change, but I don't think that
should be covered by this proposal, unless it turns out to be way too
small.


> I would like to see one more scenario covered if there is sufficient
> time. When users change some setting in their macports configuration
> (I have https://trac.macports.org/wiki/LibcxxOnOlderSystems in mind,
> but there might be other cases), it would be super helpful if users
> with a working MacPorts installation could run
> sudo port 
> which would:
> - remember which packages are installed
> - deactivate all of them (and perhaps move the files to some temporary
> location just in case they are needed again)
> - reinstall the same packages using the new settings
> 
> Basically the same scenario as with regular migration, except that the
> user would not upgrade the OS, only some of the basic macports
> settings (OK, I'm not sure how exactly you planned to implement the
> migration tool, I'm just speculating).

Good idea, that could just be a run of 'port snapshot' and 'port
migrate' without the base update part.


HTH,
-- 
Clemens

Re: GSoC'17: Add migrate action to port command Project

[Mojca Miklavec]

Hi,

I'm not too familiar with the project, but just some of my thoughts.

One of the biggest pains at the moment is the fact that we don't
record requested variants anywhere. We do record requested ports, but
not the variants.

It may easily happen that "port install foo" would install variant A
on one OS version and variant B on another. If you just record all the
variants, the user might end up with variant A installed on the new OS
even if he never requested it.

I don't know whether that fits anywhere within the scope of the
project, but I would find it super useful if one of the side products
of the project would be extension of the SQL table that would also
record +requested and -excluded variants.

It might be slightly tricky to properly cover all the corner cases.
One might run
"sudo port install foo -x11"
where "foo" depends on "bar" ("bar" will not be requested and doesn't
even have option x11) and "bar" depends on "baz" ("baz" will also not
be requested, but will have the option x11 that perhaps(?) needs to be
disabled).

That is: at least having that info available would already be a huge
benefit. If "sudo port upgrade outdated" would take that info into
account ... so much better.

I don't want to distract you from the goal of the project, but if
talking about upgrades etc, taking a bit more care about the variants
seems relatively important to me.


***


I would like to see one more scenario covered if there is sufficient
time. When users change some setting in their macports configuration
(I have https://trac.macports.org/wiki/LibcxxOnOlderSystems in mind,
but there might be other cases), it would be super helpful if users
with a working MacPorts installation could run
sudo port 
which would:
- remember which packages are installed
- deactivate all of them (and perhaps move the files to some temporary
location just in case they are needed again)
- reinstall the same packages using the new settings

Basically the same scenario as with regular migration, except that the
user would not upgrade the OS, only some of the basic macports
settings (OK, I'm not sure how exactly you planned to implement the
migration tool, I'm just speculating).


***


I cannot comment on Bash vs. Tcl, but I imagine that quite a bit
should be done in Tcl.

Some example of mixture of Bash and Tcl that come to my mind:
https://github.com/macports/mpbb
For example the code that prints variants:
https://github.com/macports/mpbb/blob/master/tools/canonical-variants.tcl

(Don't bother trying to understand what that does. Just that one can
write some Tcl script that uses MacPorts "API" even if you use a
completely different language from the outside.)


***


Just curious: how exactly does Xcode affect this project?

Mojca

Re: GSoC'17: Add migrate action to port command Project

[Umesh Singla]

Hi


> I believe at a minimum we should plan on two new actions, port “snapshot”
> and “migrate”. A snapshot will the installed ports, their variants and if
> the port was “requested”.
>
> The migrate action will call the snapshot action to create and/or use
> snapshots to rebuild ports on the new platform.
>

This looks quite a plan.
So, what snapshot action will do is to take a "snapshot" of each installed
port and keep a list of these snapshots having a name, requested boolean
and other required info (will try to figure it out soon) additionally.

A variants table will help to specify the appropriate variants, so
basically an indirect version of this: `sudo port install port_name
+variant1 +variant2`.

Migrate action will take these snapshots and use the original guidelines in
the migration docs to rebuild acc to the specifications we now have.
Correct me if I'm wrong.


> I think it is a good idea to include snapshot as an early milestone.


> Does the migrate and snapshot actions help you divide the work?
>

Yes, thanks, this gives a lot of direction to my work.


> The snapshot action is basically an inventory of what is installed along
> with meta data like requested and variants. We will store snapshots in our
> sqlite database.
>

I hope I understood it well as above.


>
> > I don't know if we'll allow Tcl to run in the new environment.
> Otherwise, it seems to implement port with a bash script calling on sqlite3
> on the macports registry/registry.d to get the relevant data.
>
> I think we can require any version of port that includes the migrate
> action, I do not think we will need bash. Clemens, others, do you concur?
>

That time, I was trying to say that for a more general scenario, it'll take
to implement port with a bash front-end that invoked sqlite3 directly upon
the MacPorts registry (located in ${prefix}/var/macports/registry/registry.d)
(not anymore) and extracting the required data. Yes, the above seems an
elegant solution but still migrate action needs more implementation
detailing. Let me work on it.

> Also, will the project phasing out dependency on XCode tools affect this
> one anytime soon?
> Good question, I cannot give you a definitive answer at the moment but we
> should keep this in mind.
>

will include this as stretch goal or something like "Plans after GSoC" :D

An additonal point of thought would be to resolve the conflicts arising if
there are conflicting ports in the list and I guess, order of reinstalling
ports could also be one such factor for conflict if not the default
variants?

Thanks,
Umesh

Re: GSoC'17: Add migrate action to port command Project

[Bradley Giesbrecht]

> On Mar 28, 2017, at 11:55 AM, Umesh Singla  wrote:
> 
> Hi Brad,
> 
> I'm UTC +5:30. Around 10:00 to 15:00 (UTC -8) is a good time, I suppose. 

Ok, good, you make it easy for me :)

> I have been going through discussion on the mailing list again and found that 
> the automating the tasks given under Migration documentation is just a start 
> for the port migration tool we need. Can you tell me what other cases it 
> needs to cater?

I believe at a minimum we should plan on two new actions, port “snapshot” and 
“migrate”. A snapshot will the installed ports, their variants and if the port 
was “requested”.

The migrate action will call the snapshot action to create and/or use snapshots 
to rebuild ports on the new platform.

> Do I need to include the solution proposals to them in my GSoC proposal at 
> this time or we can work it out as we go?

I think it is a good idea to include snapshot as an early milestone.

> As of now, I'm not able to divide the work into different phases, might need 
> your help.

Does the migrate and snapshot actions help you divide the work?
The snapshot action is basically an inventory of what is installed along with 
meta data like requested and variants. We will store snapshots in our sqlite 
database.

It is early on but I can imagine a schema like:

Tables
snapshots (
id,
datetime,
note
)
snapshot_ports (
id,
snapshots_id,
port_name,
requested
)
snapshot_port_variants (
id,
snapshot_ports_id,
variant_sign, // (we allow plus and minus variants like +variant1 
-variant2)
variant_name
)

> I don't know if we'll allow Tcl to run in the new environment. Otherwise, it 
> seems to implement port with a bash script calling on sqlite3 on the macports 
> registry/registry.d to get the relevant data. 

I think we can require any version of port that includes the migrate action, I 
do not think we will need bash. Clemens, others, do you concur?

> Also, will the project phasing out dependency on XCode tools affect this one 
> anytime soon?

Good question, I cannot give you a definitive answer at the moment but we 
should keep this in mind.

—
Brad

Re: GSoC'17: Add migrate action to port command Project

[Bradley Giesbrecht]

> On Mar 23, 2017, at 9:08 PM, Jackson Isaac  wrote:
> 
> Hi Umesh,
> 
> 
> First of all, Welcome to The MacPorts Project. We are always happy to
> see new developers around.

Yes, welcome.

> On Fri, Mar 24, 2017 at 4:30 AM, Umesh Singla  wrote:
>> Hello,
>> 
>> As we discussed on IRC, I've been looking into the project "Add migrate
>> action to port command" and hoping to submit a draft proposal very soon.
>> I'll need your help in putting up a nice proposal for sure.
>> 
> 
> Submitting a draft early is a good idea so that we could guide you
> well in time. We had submitted a template for proposal on the
> summerofcode.withgoogle.com page. Let me know if you are unable to
> access it.
> 
>> I'm currently trying to acquaint myself with the MacPorts codebase and came
>> across the Macports guide [0]. Thanks for keeping such guides for newbies.
>> The project description points to a migration instruction guide [1],
>> automating which is more or less the goal of the project, am I right?
>> 
>> Since final submission deadline is close, I'll be grateful if someone can
>> point me in the right direction for now and not waste time fiddling around.
>> 
>> Also, @Jackson, I'm not able to find any discussion about this project on
>> mailing list threads. Can you link me to them? I think they'll help me in
>> understanding the need and expected outcomes by the community from this
>> project and might give a better insight into the project.
>> 
> 
> It was in macports-users (I thought it was posted on macports-dev,
> took me some time to search :P)
> https://lists.macports.org/pipermail/macports-users/2017-March/042837.html
> 
>> @pixilla, if we need to discuss, what'd be the best time according to your
>> timezone?
>> 
> 
> Pixilla,
> 
> Could you please guide the student on migration tool project.

Yes. I am short on time today. Umesh , what timezone are you and what hours are 
you generally available for online chat or IRC? I am UTC-8.
If we  can set some standard times to communicate that would be a good start.

> Let us know if you face any issue(s).

Jackson, will do, thank you for your leadership.

—
Brad

Re: GSoC'17: Add migrate action to port command Project

[Jackson Isaac]

Hi Umesh,


First of all, Welcome to The MacPorts Project. We are always happy to
see new developers around.

On Fri, Mar 24, 2017 at 4:30 AM, Umesh Singla  wrote:
> Hello,
>
> As we discussed on IRC, I've been looking into the project "Add migrate
> action to port command" and hoping to submit a draft proposal very soon.
> I'll need your help in putting up a nice proposal for sure.
>

Submitting a draft early is a good idea so that we could guide you
well in time. We had submitted a template for proposal on the
summerofcode.withgoogle.com page. Let me know if you are unable to
access it.

> I'm currently trying to acquaint myself with the MacPorts codebase and came
> across the Macports guide [0]. Thanks for keeping such guides for newbies.
> The project description points to a migration instruction guide [1],
> automating which is more or less the goal of the project, am I right?
>
> Since final submission deadline is close, I'll be grateful if someone can
> point me in the right direction for now and not waste time fiddling around.
>
> Also, @Jackson, I'm not able to find any discussion about this project on
> mailing list threads. Can you link me to them? I think they'll help me in
> understanding the need and expected outcomes by the community from this
> project and might give a better insight into the project.
>

It was in macports-users (I thought it was posted on macports-dev,
took me some time to search :P)
https://lists.macports.org/pipermail/macports-users/2017-March/042837.html

> @pixilla, if we need to discuss, what'd be the best time according to your
> timezone?
>

Pixilla,

Could you please guide the student on migration tool project.

Let us know if you face any issue(s).

-- 
Jackson Isaac

GSoC'17: Add migrate action to port command Project

[Umesh Singla]

Hello,

As we discussed on IRC, I've been looking into the project "Add migrate
action to port command" and hoping to submit a draft proposal very soon.
I'll need your help in putting up a nice proposal for sure.

I'm currently trying to acquaint myself with the MacPorts codebase and came
across the Macports guide [0]. Thanks for keeping such guides for newbies.
The project description points to a migration instruction guide [1],
automating which is more or less the goal of the project, am I right?

Since final submission deadline is close, I'll be grateful if someone can
point me in the right direction for now and not waste time fiddling around.

Also, @Jackson, I'm not able to find any discussion about this project on
mailing list threads. Can you link me to them? I think they'll help me in
understanding the need and expected outcomes by the community from this
project and *might give a better insight into the project*.

@pixilla, if we need to discuss, what'd be the best time according to your
timezone?

Hoping for a quick reply,
Umesh

[0]: https://guide.macports.org/
[1]: https://trac.macports.org/wiki/Migration