Re: package upgrade strategy

[r0ller]

Hi All,

Thanks again for all the answers in the topic! I read all of them but I cannot 
answer each in separate emails so I just picked the latest to continue the 
thread.

As earlier mentioned I agree with the advantages of the pkgsrc based solutions 
and in the meantime I also found a nice summary of the different approaches at 
https://wiki.netbsd.org/pkgsrc/how_to_upgrade_packages/ (better later than 
never). However, I pursued the issue further to get a workaround (till I have 
time to rebuild things via pkgsrc) with the binary packages and I think I found 
something based on which I could build something.

I only tried it manually and it seems to work but of course in case of more 
pkgs a script would be better to manage all this:
-in case a package needs to be updated to a newer version, copy its shared libs 
to a specific directory (say soold)
-record the old pkg name (together with the one that required updating it) in 
an sqlite db and the files copied to the soold directory
-add soold directory to LD_LIBRARY_PATH if it's not yet added

The solution would be better if it was somehow possible to revert the changes 
i.e. if I don't like the newly installed stuff, i.e. just downgrade the 
packages to their earlier versions which could be figured out from the db file 
that keeps track of the removed pkgs. However, afaik binary pkg downgrade is 
not possible either pkgin or by the pkg_* tools.

I guess this is against quite a many things and how they're meant to be done in 
NetBSD but as a workaround this may come handy. What do you think? Can anyone 
forsee already a "design flaw" or whatever that'd prevent this from being 
feasible?

Thanks & regards,
r0ller


 Eredeti levél 
Feladó: Greg Troxel < g...@lexort.com (Link -> mailto:g...@lexort.com) >
Dátum: 2017 szeptember 30 18:09:06
Tárgy: Re: package upgrade strategy
Címzett: r0ller < r0l...@freemail.hu (Link -> mailto:r0l...@freemail.hu) >
 
pkgsrc is designed, more or less, to have packages built consistently
from the same source tree. You can often get away with having some
packages be old. Other than the old packages not depending on or being
dependencies of newer ones, you're on thin ice.
If a package is not available in a newer branch, that could be because
it was removed from pkgsrc, or because it didn't build. You can check
out the sources and have a look. You can also look at the mail archives
for pkgsrc-bulk and see what happened, and find the build logs. It may
be that midori builds on NetBSD 8 (not yet out) but not 7, or 7 but not
6. This can be about the versions of X things in the base system, or it
can be about compilers. So looking at the build logs for various
systems can be useful.
pkgsrc does not require you to choose between building from source and
binary packages. But, you should use a source tree consistent with the
binary packages you are using. Right now I think the latest binary
packages for NetBSD are from 2017Q2. 2017Q3 exists in source form (and
I'm behind on drafting an announcement), but NetBSD's package building
machines are down due to data center work (we are grateful guests). I
expect binary packages within a few weeks.
Also, you can build your own binary packages, and use them with pkgin.
You just have to make a summary file, which is easy.
pkg_info -X *gz | bzip2 > pkg_summary.bz2
So if you have a fast machine, you can build, and install them on the
slower machine (just rsync them and add a repositories.conf line).
Packages without active upstreams are harder to maintain and can tend to
run into problems. Midori's last release appears to be from August of
2015. That's not really really old, but it's long enough to suggest a
possible issue. If you want to run an older build of midori, it's going
to expect dependencies that are ABI-compatible, and this rapidly gets
difficult.
Overall, I think your best (but not easiest) path to success will be to
use 2017Q3 sources, update everything you can via the forthcoming binary
packages, and then to build midori from source, perhaps fixing the
build. We would then want to commit the fixes, and for anything that
belongs upstream to be comitted there and then have a new release. So
far it looks like the patches in pkgsrc for midori are very minor and
not obviously appropriate for upstream.
 

Re: package upgrade strategy

[Greg Troxel]

pkgsrc is designed, more or less, to have packages built consistently
from the same source tree.  You can often get away with having some
packages be old.  Other than the old packages not depending on or being
dependencies of newer ones, you're on thin ice.

If a package is not available in a newer branch, that could be because
it was removed from pkgsrc, or because it didn't build.  You can check
out the sources and have a look.  You can also look at the mail archives
for pkgsrc-bulk and see what happened, and find the build logs.  It may
be that midori builds on NetBSD 8 (not yet out) but not 7, or 7 but not
6.  This can be about the versions of X things in the base system, or it
can be about compilers.  So looking at the build logs for various
systems can be useful.

pkgsrc does not require you to choose between building from source and
binary packages.  But, you should use a source tree consistent with the
binary packages you are using.  Right now I think the latest binary
packages for NetBSD are from 2017Q2.  2017Q3 exists in source form (and
I'm behind on drafting an announcement), but NetBSD's package building
machines are down due to data center work (we are grateful guests).  I
expect binary packages within a few weeks.

Also, you can build your own binary packages, and use them with pkgin.
You just have to make a summary file, which is easy.

  pkg_info -X *gz | bzip2 > pkg_summary.bz2

So if you have a fast machine, you can build, and install them on the
slower machine (just rsync them and add a repositories.conf line).

Packages without active upstreams are harder to maintain and can tend to
run into problems.  Midori's last release appears to be from August of
2015.  That's not really really old, but it's long enough to suggest a
possible issue.  If you want to run an older build of midori, it's going
to expect dependencies that are ABI-compatible, and this rapidly gets
difficult.

Overall, I think your best (but not easiest) path to success will be to
use 2017Q3 sources, update everything you can via the forthcoming binary
packages, and then to build midori from source, perhaps fixing the
build.  We would then want to commit the fixes, and for anything that
belongs upstream to be comitted there and then have a new release.  So
far it looks like the patches in pkgsrc for midori are very minor and
not obviously appropriate for upstream.



signature.asc
Description: PGP signature

Re: package upgrade strategy

[Robert Elz]

Date:Thu, 28 Sep 2017 22:20:50 +0200 (CEST)
From:r0ller 
Message-ID:  


  | but I hoped that there's a way to avoid compiling relatively big programs
  | (like seamonkey) in a case when I just want to check something out [...]

There is - you can keep using the binary packages, for everything that
still exists in pkgsrc (and for which binary packages are available).
You just need to use the sources for midori (or any other package that
has been deleted) because there are no new binary packages being produced
for that.

For everything else you just do binary package upgrades, then when
everything is in place (which will have resulted in midori having
been deleted) you just compile your copy of that (using the old pkgsrc
directory for it) from sources.

As long as it continues to build, that should be easy enough.  Of
course, over time, that gets less and less likely, unless you are
upgrading it yourself.

If you want to protect it so you have a binary that will work like it
does now, regardless of anything else that happens, you should be able
to make that work, by setting up an entirely separate /usr/pkg tree (ie:
storing it someplace else) - get a version of pkgsrc that includes midori
(an older one than current) change the PKG_DBDIR and LOCALBASE / PREFIX,
and build just midori from source with those settings (which will also
build all of its dependencies, since in that tree, nothing will exist yet).

Once that's done, simply leave that tree alone forever, and never touch
it again.   It will remain completely independent of anything that is
normally done with pkgsrc, not influencing it at all.  In the regular
/usr/pkg you just keep doing binary pkg_add pkg_delete ... as desired
to install and delete packages you want to test.

kre

ps: "kids" is perfectly OK to use at least for this native English speaker.

Re: package upgrade strategy

[r0ller]

Hi All,

Thanks for all your answers (Alistair, Brad, Jeff, Jeremy)! They all seem to 
point in one direction i.e. to use pkgsrc with which I don't have any problem. 
I'm kind of used to it but I hoped that there's a way to avoid compiling 
relatively big programs (like seamonkey) in a case when I just want to check 
something out and get rid of it if it does not fit the bill. The scenario is 
something like that I have a 10 years old desktop for our kids (some say this 
word is negative for native English speakers, I'm not one of them so please, 
forgive) to play online flash games, watch youtube videos, etc. with 3gb ram, 
intel core2 conroe cpu with 1 core+igp and a conroe asrock board which all 
serves pretty well at their age for this purpose -at least with NetBSD;) Midori 
did fit the bill for everything they wanted till they bumped into codecombat 
which doesn't work. So I wanted to quickly check out another browser and this 
is exactly when I don't have time to compile anything:) As mentioned, by 
checking out the earlier 7.0 Q1 repo helped with seamonkey and codecombat runs 
fine in that but in such cases compiling each browser till I find the right one 
is not an option. I mean, before seamonkey I also checked out epiphany and 
firefox, which have the same bug as midori in that online game. Finding out 
that seamonkey fits the bill took roughly 15 mins, but compiling three browsers 
would have taken a lot more than that. Don't get me wrong, in general I agree 
with all your suggestions, I just hoped that there's still a way to keep 
different versions of pkgs in parallel.

Best regards,
r0ller

 Eredeti levél 
Feladó: Alistair Crooks < a...@pkgsrc.org (Link -> mailto:a...@pkgsrc.org) >
Dátum: 2017 szeptember 28 18:20:31
Tárgy: Re: package upgrade strategy
Címzett: Jeff_W < j...@sdf.org (Link -> mailto:j...@sdf.org) >
 
On 28 September 2017 at 08:40, Jeff_W <j...@sdf.org> wrote:
> % cd /pkgsrc/foo/pkg-A
> % sudo make clean && sudo make replace
>
> If the above breaks pkg-C:
>
> % cd /pkgsrc/foo/pkg-C
> % sudo make clean && sudo make clean-depends && sudo make update
Orthogonal to this discussion - pkgsrc was modified to use
just-in-time su in the early 2000s, and avoids interesting issues like
fetching sources as root. You are definitely encouraged to use it.
If you'd prefer to use sudo, rather than su, put this in your etc/mk.conf:
.if exists(${LOCALBASE}/bin/sudo)
SU_CMD= ${LOCALBASE}/bin/sudo /bin/sh -c
.endif
Best,
Alistair

Re: package upgrade strategy

[coypu]

I suspect this is more fallout from people mis-updating libraries.
try manually deleting and adding midori again. I'm hoping it rebuilt the
package even though there's no indication to the binary package bits
that it changed.

Re: package upgrade strategy

[Brad Spencer]

r0ller  writes:

> Hi Brad,
>
> I'd do that if it was possible (having the newest versions of all my 
> packages) but as said, midori is not available any more in the 7.1 package 
> repo so it cannot be upgraded (not even the same version of midori is 
> available with just an updated list of dependencies) and if I upgrade icu (to 
> version 59), midori starts complaining about not finding the earlier version 
> of it (version 58). What if I just used pkg_add for icu-59? Or if I mark 
> icu-58 non-autoremovable by pkgin, would the install of icu-59 leave icu-58 
> untouched?
>
> Best regards,
> r0ller
>


Ya, you are in a bad situation that will likely never allow you to
update using anything resembling "normal" using prebuilt binary
packages.  I.e.  no more pkgin anymore from a remote repo.

You are facing the requirement to build packages yourself from source
and the need to create a consistant "bundle" of packages for a system.
You may be able to resolve the issue by creating a local midori package
and build from that, for example...

I have never used the prebuilt binary packages, and have always built
and rebuilt from source due to various complicated reasons for more than
15 years.  It has allowed me to do things like include a local version
of a package that isn't "out yet" in pkgsrc and has allowed me to use
older packages that have been removed from pkgsrc [or updated to a
version I didn't want] as long as they would be ok with the dependent
packages in pkgsrc.

I could describe what I am doing if you are interested and want to do
this sort of thing.  It honestly isn't hard and mostly just requires a
bit of organization and some disk space.



-- 
Brad Spencer - b...@anduin.eldar.org - KC8VKS
http://anduin.eldar.org  - & -  http://anduin.ipv6.eldar.org [IPv6 only]

Re: package upgrade strategy

[Jeremy C. Reed]

On Thu, 28 Sep 2017, r0ller wrote:

> By the way, what kind of difference is indicated by the number in the 
> 'nb' suffix?

Means the original code (upstream source) was not changed. The nb means 
we may be building or installing it differently (like due to a new 
patch, new build option, or a dependency change, for example).

> Another question would be if it's possible to keep different 
> versions of a package installed? I know in case of shared libs it may 
> be tricky because of the symlinks but the runtime linker is not 
> looking for the symlink I hope but the versioned soname, right? Any 
> hints are welcome!

Well if you build your own from pkgsrc, you can use a different 
PKG_DBDIR and LOCALBASE. (Maybe bootstrap it with different settings.)
But then you have an additional install to manage and lots of resources
potentially wasted.

(Sorry you had a package disappear.)

Re: package upgrade strategy

[Jeff_W]

Alistair Crooks  wrote:

> On 28 September 2017 at 08:40, Jeff_W  wrote:
> > % cd /pkgsrc/foo/pkg-A
> > % sudo make clean && sudo make replace
> >
> > If the above breaks pkg-C:
> >
> > % cd /pkgsrc/foo/pkg-C
> > % sudo make clean && sudo make clean-depends && sudo make update
>
> Orthogonal to this discussion - pkgsrc was modified to use
> just-in-time su in the early 2000s, and avoids interesting issues like
> fetching sources as root. You are definitely encouraged to use it.
>
> If you'd prefer to use sudo, rather than su, put this in your etc/mk.conf:
>
> .if exists(${LOCALBASE}/bin/sudo)
> SU_CMD= ${LOCALBASE}/bin/sudo /bin/sh -c
> .endif

Nice; wasn't aware of that - thanks Alistair

Re: package upgrade strategy

[Alistair Crooks]

On 28 September 2017 at 08:40, Jeff_W  wrote:
> % cd /pkgsrc/foo/pkg-A
> % sudo make clean && sudo make replace
>
> If the above breaks pkg-C:
>
> % cd /pkgsrc/foo/pkg-C
> % sudo make clean && sudo make clean-depends && sudo make update

Orthogonal to this discussion - pkgsrc was modified to use
just-in-time su in the early 2000s, and avoids interesting issues like
fetching sources as root. You are definitely encouraged to use it.

If you'd prefer to use sudo, rather than su, put this in your etc/mk.conf:

.if exists(${LOCALBASE}/bin/sudo)
SU_CMD= ${LOCALBASE}/bin/sudo /bin/sh -c
.endif

Best,
Alistair

Re: package upgrade strategy

[Jeff_W]

> I hope someone can give an answer to this question from a newbie: how can
> I handle situations when an already installed package (say A) needs to be
> upgraded due to a newly installed package (say B) as it's being a
> dependency for it but another already installed package (say C) still
> needs the older version of package A? Upgrading package C may be thought
> of as help but it disappeared from the 7.1 package repo and as far as I
> could see it was last present in 7.0_2017Q1.

% cd /pkgsrc/foo/pkg-A
% sudo make clean && sudo make replace

If the above breaks pkg-C:

% cd /pkgsrc/foo/pkg-C
% sudo make clean && sudo make clean-depends && sudo make update

HTH,
jgw

Re: package upgrade strategy

[r0ller]

Hi Brad,

I'd do that if it was possible (having the newest versions of all my packages) 
but as said, midori is not available any more in the 7.1 package repo so it 
cannot be upgraded (not even the same version of midori is available with just 
an updated list of dependencies) and if I upgrade icu (to version 59), midori 
starts complaining about not finding the earlier version of it (version 58). 
What if I just used pkg_add for icu-59? Or if I mark icu-58 non-autoremovable 
by pkgin, would the install of icu-59 leave icu-58 untouched?

Best regards,
r0ller

 Eredeti levél 
Feladó: Brad Spencer < b...@anduin.eldar.org (Link -> 
mailto:b...@anduin.eldar.org) >
Dátum: 2017 szeptember 28 14:13:53
Tárgy: Re: package upgrade strategy
Címzett: r0ller < r0l...@freemail.hu (Link -> mailto:r0l...@freemail.hu) >
 
r0ller <r0l...@freemail.hu> writes:
> Hi All,
>
> I hope someone can give an answer to this question from a newbie: how can I 
> handle situations when an already installed package (say A) needs to be 
> upgraded due to a newly installed package (say B) as it's being a dependency 
> for it but another already installed package (say C) still needs the older 
> version of package A? Upgrading package C may be thought of as help but it 
> disappeared from the 7.1 package repo and as far as I could see it was last 
> present in 7.0_2017Q1.
>
> If anyone is interested in the details, package A is icu, package B is 
> seamonkey and package C is midori. What I did for now was that I changed 
> /usr/pkg/etc/pkgin/repositories.conf so that it points to 7.0_2017Q1 and 
> installed seamonkey from that repo as the versions are not that different 
> (seamonkey-2.46nb7 in 7.0_2017Q1 and seamonkey-2.46nb8 in 7.1) and it left 
> icu untouched. By the way, what kind of difference is indicated by the number 
> in the 'nb' suffix? Another question would be if it's possible to 
> keep different versions of a package installed? I know in case of shared libs 
> it may be tricky because of the symlinks but the runtime linker is not 
> looking for the symlink I hope but the versioned soname, right? Any hints are 
> welcome!
>
> Best regards,
> r0ller
Hello...
I have been running pkgsrc for a very long time and you won't like my
answer
Basically, you don't try to do this with piecemeal updates.
It honestly will be simpler to delete all of the packages, saving
configs first, and reinstall the new versions. This may sound quite
extreme, but if you have the situation you describe there will be too
many other interdependencies to make anything else reasonable and sane,
especially in the long run.
There are techniques to make this much less harsh then it would be
otherwise, and I can tell you about them if you want, but I really
suspect you will find it much simpler to do as I suggested.
--
Brad Spencer - b...@anduin.eldar.org - KC8VKS
http://anduin.eldar.org - & - http://anduin.ipv6.eldar.org [IPv6 only]