Re: Is pkg quarterly really needed?

[Mel Pilgrim]

On 04/19/2017 22:22, Mark Linimon wrote:

On Wed, Apr 19, 2017 at 04:37:05PM -0400, scratch65...@att.net wrote:

(Right now, it's quite hard to resist the paranoid suspicion that
maybe this crazy, anti-real-user behavior is a subtle way to kill
freebsd altogether by driving away the non-hobbyists.)


That's one explanation.

The other, possible, explanation, is that the efforts of a group of
volunteers isn't adequate enough for every use case -- including your own.

But, of course, feel free to cast aspersions wherever and whenever.  We're
just machines, we have no feelings whatsoever.

Now if no one minds, I'm going to go back to contemplate the existential
question of "why do I even bother trying to fix things".


Because the work you and everyone else does on FreeBSD makes it possible 
for me to do the work I do for dozens of non-profits who need safe, 
reliable network and internet services without paying retail or being 
pigeonholed into an online provider's one-size product.


For every scratch65535, there's one like me who doesn't have to deal 
with the expense of Windows servers or the increasingly black-box nature 
of Linux distros without also having to compile everything from scratch 
on the weekly.


Mel, the IT admin who gets to run 11-R and -CURRENT in exchange for free 
books and actually helping people because she doesn't have to say things 
like, "I can do that, but the licensing is half your 5-year budget."

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Kurt Jaeger]

Hi!

> >> If the whole repository builds doesn't it mean by default that any
> >> subset also builds?

> > If we defined a repo build only as valid if everything builds,
> > the whole repo is never valid, because approx. 10% of
> > the ports tree breaks at any given time. More, if you add options.

> That's an interesting observation, I didn't know that. Does it mean that 
> the quarterly port tree is no better or worse than the main branch?

It's a little bit better, I guess. Oh, indeed, I erred with my numbers.

Here's the link I'm looking at right now:

http://beefy9.nyi.freebsd.org/jail.html?mastername=110amd64-default

It has stats on the full ports builds for 11.0-amd64.

Approx. 26000 ports, approx. 50 failed to build, 70 skipped, 300+ ignored.

So it looks much better than I thought, but not 100%.

> And is any tree ever build with non-default options?

I don't know. I build my own repos with these settings:

DEFAULT_VERSIONS= perl5=5.24 python=2.7 python3=3.6 ruby=2.3 pgsql=9.6 php=7.1 
mysql=10.1m gcc=6

And I use some non-default options, but not that many.

> >> My assumption was that only version
> >> upgrades are progressed from CURRENT to STABLE to RELEASE.

> > Leads to a stagnating tree downstream, if you find maintainers for it.
> > That's the model Debian is using, and it has other issues. [...]

> Well, they can't be as unhappy as, say, Centos, where packages are 
> really old. Also, I bet not all users are unhappy when the ports are not 
> updated quickly. Corporate users tend to prefer stable versions even if 
> they are getting a bit old, enthusiasts tend to prefer newest versions. 

Hmm, our company isn't big, but keeping up with the security
stuff needs to happen, anyway, so we mostly use recent pkg trees.
Like the saying goes: Update early, update often, automate updates.

Incremental learnings/breakings are easier to handle than huge
across-the-board upgrades.

That's my experience after almost 30 years in that field.
Believe me, I already tried other approaches 8-} I still
have a FreeBSD 4.11 box in my fleet to care about, where
I recently updated OpenSSH and bind 8-}

> FreeBSD can't cater for both groups a the same time.

Here I disagree! Right now, ports-HEAD surely *does* cater for
the upgrade-junkies 8-}, but this also allows very fast innovation
cycles.

The quarterly trees are a first step to test how stability can also
be provided. We're still learning.

> Which group has been chosen, if it has been chosen?

It's also an ecosystem thing. If the 'stable at a price' niche
is covered by Debian etc, FreeBSD in the past needed to find
another niche. But it's not exclusivly so, so the quarterly tree
was created. The first was 2014Q1, so we're 14 iterations into
this experiment. Still plenty of things to learn.

> Are we defaulting to enthusiasts?

I don't think so! We're doing what we can to cover all use-cases.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Grzegorz Junka]

On 21/04/2017 02:51, Kurt Jaeger wrote:



If the whole repository builds doesn't it mean by default that any
subset also builds?

If we defined a repo build only as valid if everything builds,
the whole repo is never valid, because approx. 10% of
the ports tree breaks at any given time. More, if you add options.


That's an interesting observation, I didn't know that. Does it mean that 
the quarterly port tree is no better or worse than the main branch? And 
is any tree ever build with non-default options? If no, how do you know 
how many are failing in that case?



My assumption was that only version
upgrades are progressed from CURRENT to STABLE to RELEASE.

Leads to a stagnating tree downstream, if you find maintainers for it.
That's the model Debian is using, and it has other issues. Especially
the load for the maintainers is huge, and users are unhappy
that the packages are getting old. Debian has approx. 6 times
more committers than we have, when I last looked, and more maintainers.

If we take from that that we have to grow our committer base, yes.
Can we reason that unless we have that base, we can't follow that
model ? Maybe.


Well, they can't be as unhappy as, say, Centos, where packages are 
really old. Also, I bet not all users are unhappy when the ports are not 
updated quickly. Corporate users tend to prefer stable versions even if 
they are getting a bit old, enthusiasts tend to prefer newest versions. 
FreeBSD can't cater for both groups a the same time. Which group has 
been chosen, if it has been chosen? Are we defaulting to enthusiasts?


Grzegorz
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Kurt Jaeger]

Hi!

> >> I understand that the main problem with quarterly branches is that they
> >> start from an unstable edge and mature with time, then after three
> >> months at the most mature point they are being deleted and replaced with
> >> a new unstable edge. So, there is no good point of reference to use as a
> >> source of packages.
[how stable the repos appear for the users]

> I hardly remember a single upgrade to 
> the longest set that didn't result in some packages being broken. [...]
> most of those breakages resulted from me selecting non-standard options 

Ah, options. Leads to exploding complexity.

> So, my experience is quite different to yours...

Indeed.

> If the whole repository builds doesn't it mean by default that any 
> subset also builds?

If we defined a repo build only as valid if everything builds,
the whole repo is never valid, because approx. 10% of
the ports tree breaks at any given time. More, if you add options.

> My assumption was that only version 
> upgrades are progressed from CURRENT to STABLE to RELEASE.

Leads to a stagnating tree downstream, if you find maintainers for it.
That's the model Debian is using, and it has other issues. Especially
the load for the maintainers is huge, and users are unhappy
that the packages are getting old. Debian has approx. 6 times
more committers than we have, when I last looked, and more maintainers.

If we take from that that we have to grow our committer base, yes.
Can we reason that unless we have that base, we can't follow that
model ? Maybe.

> On the other hand developers would be more inclined to do changes in 
> CURRENT if they know that they are not going to break ports for the 
> majority of users who should use STABLE or RELEASE.

This fear to break something is not a big issue in general.
This is covered by build-tests using poudriere, most of the time.

It is an issue for those ports where lots of things depend on a
port, because of changes to that port that lead to cascading
failures.

> > It's just that asking the team that's barely keeping up
> > to do that *on top*, that will probably not work.

> That was supposed to be more like *instead* rather than *on top*.

As long as we're not plundering countries, we normally do not burn
the ships until it's safe to proceed 8-}

> >> From that short description it should be more or less
> >> obvious if that approach is better/doable when opposed to quarterly
> >> branches?

> > There's a way to find out: Try it.

> Not the best way TBH. I would rather hear some opinions first as I am 
> sure there are plenty of conditions and requirements I haven't even 
> imagined myself yet.

It's a good approach to learn about the requirements etc. Most
of the knowledge is, from what I understand, not documented
in a way that helps to write a requirements document. It's in
the code of the currently running system, like layers of
ashes in old cities. Reading those layers requires a certain mindset.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Grzegorz Junka]

Hi :)

On 20/04/2017 19:57, Kurt Jaeger wrote:

Hi!


I understand that the main problem with quarterly branches is that they
start from an unstable edge and mature with time, then after three
months at the most mature point they are being deleted and replaced with
a new unstable edge. So, there is no good point of reference to use as a
source of packages.

First, let me say that for my use cases, the pkg tree is in consistent
state most of the time. We have a set of approx. 2500 pkg's build etc,
so roughly 10% of the full repo. We update the ports tree and build the
repo every week, just to see if all is fine, and we update
and build, if we need a package we do not yet have in our repo.
For the thrills, look at our repo at https://repo.nepustil.net


I am maintaining four sets of pkg's for my desktop, laptop and server 
(in two variants) - anything between 100 to 1300 pkg's resulted from 
selecting around 10-200 ports depending on the set. I usually first 
build for my desktop, then if everything is fine I rebuild the set for 
my laptop, and finally the server. I hardly remember a single upgrade to 
the longest set that didn't result in some packages being broken. Sure, 
most of those breakages resulted from me selecting non-standard options 
(but if I wanted standard options then I would simply install from the 
official distribution, wouldn't I?). Only last weekend I filled two bugs 
against ports that didn't compile when either non-default options were 
selected in those packages or in some of their dependent packages.


So, my experience is quite different to yours...


Hmm, let's try this thought experiment.

For each package, keep the whole repo when it was last build sucessfully
(Note: we're not yet discussing whether it runs!)

Worst case: We have n ports and n repos. The question is:
what would be the average case ? Would that be a sustainable model ?

Now, the next question: Even if we have all the repos,
would we find *one* repo where *two* packagew we'd like to have
are in a consistent (build-!)state ? What about the 250+
that are normally needed for a small server ? Or the 1200+
for a multi-role server (some will say: "nah, don't do it, use
container/jails/ whatever virtualisiation").


If the whole repository builds doesn't it mean by default that any 
subset also builds? All packages are build incrementally, the packages 
that don't depend on any other packages are build first, then any 
packages that depend on them, and so on. Sure, it doesn't mean that they 
also run properly, but that's a different story. If packages are build 
then people can install them and fill bugs. That would be the reason for 
having CURRENT, STABLE and RELEASE where fixes would be gradually 
progressed between them.



What I described is taking the good points (maturing the code through
progressing version upgrades from CURRENT, through STABLE to RELEASE)

Now, if we have ports-HEAD, and changes to that (especially fixes and
features to the ports infrastructure), it's getting more and more difficult
to backport fixes from ports-HEAD to the ports-STABLE versions, because
those do not contain dependencies and infrastructure changes.
If we backport those, we have to roll forward some other
changes. This gets out of hand very quickly.


I see where you are getting at. My assumption was that only version 
upgrades are progressed from CURRENT to STABLE to RELEASE. If a port 
can't be progressed because it requires infrastructure change then, 
well, it won't. STABLE and then RELEASE are meant to be more stable so 
we would prefer older versions that work rather than new versions that 
break. Infrastructure changes would have to be progressed eventually but 
that could be done in batches where most fixes in more edge branches 
have been fixed.



So:

If we take the sum of the brain time our maintainers and committers
deliver to keep HEAD in a moving (different from a stagnating) state,
and try to estimate how much *more* time would be needed to
keep additional trees in working conditions, only updating
what is needed, under the assumption that infrastructure changes
*need* to happen ? What would that additional brain time be ?
My inituition tells me that it would probably break the model.


On the other hand developers would be more inclined to do changes in 
CURRENT if they know that they are not going to break ports for the 
majority of users who should use STABLE or RELEASE. According to the 
principle "Failing by design 
". They would be also more 
confident when porting changes to more stable branches knowing that they 
have been tested by many users and if something could have gone wrong 
most likely already did.




Now, if someone wants to *experiment* with that, we
already have quite a few people doing that: By setting up
our own repo boxes, where we build the trees to our liking.


I am not trying to solve the problem for myself. I already have my own 
repo box. I 

Re: Is pkg quarterly really needed?

[Kurt Jaeger]

Hi!

> I understand that the main problem with quarterly branches is that they 
> start from an unstable edge and mature with time, then after three 
> months at the most mature point they are being deleted and replaced with 
> a new unstable edge. So, there is no good point of reference to use as a 
> source of packages.

First, let me say that for my use cases, the pkg tree is in consistent
state most of the time. We have a set of approx. 2500 pkg's build etc,
so roughly 10% of the full repo. We update the ports tree and build the
repo every week, just to see if all is fine, and we update
and build, if we need a package we do not yet have in our repo.
For the thrills, look at our repo at https://repo.nepustil.net

Hmm, let's try this thought experiment.

For each package, keep the whole repo when it was last build sucessfully
(Note: we're not yet discussing whether it runs!)

Worst case: We have n ports and n repos. The question is:
what would be the average case ? Would that be a sustainable model ?

Now, the next question: Even if we have all the repos,
would we find *one* repo where *two* packagew we'd like to have
are in a consistent (build-!)state ? What about the 250+
that are normally needed for a small server ? Or the 1200+
for a multi-role server (some will say: "nah, don't do it, use
container/jails/ whatever virtualisiation").

> What I described is taking the good points (maturing the code through 
> progressing version upgrades from CURRENT, through STABLE to RELEASE) 

Now, if we have ports-HEAD, and changes to that (especially fixes and
features to the ports infrastructure), it's getting more and more difficult
to backport fixes from ports-HEAD to the ports-STABLE versions, because
those do not contain dependencies and infrastructure changes.
If we backport those, we have to roll forward some other
changes. This gets out of hand very quickly.

Do we need frequent infrastructure changes ? Yes, because right now
we would not be able to build the tree, if we did not change
some knobs to cope with the newest craziness that upstreams
throw at us. We repeatedly needed relevant infrastructure changes just
to keep up with that outside world. I'm still speechless that
tz@ got php71 into the tree without so few defects. Or the parallel
mysql-variants. We still fail to provide a working maven-mechanism.
Or go-dependencies. Think of it, go apps more and more bring
their own dependencies, because that is somehow unsolved in the go
world. Some folks worked wonders with the multi-arch and cross-build
things. I can build ARM pkg repos on my amd64, that is unheard
of in most parts of the IT universe 8-} We are almost tracking
firefox, even after they added a new language (rust) to their
infrastructure.

So:

If we take the sum of the brain time our maintainers and committers
deliver to keep HEAD in a moving (different from a stagnating) state,
and try to estimate how much *more* time would be needed to
keep additional trees in working conditions, only updating
what is needed, under the assumption that infrastructure changes
*need* to happen ? What would that additional brain time be ? 
My inituition tells me that it would probably break the model.

Now, if someone wants to *experiment* with that, we
already have quite a few people doing that: By setting up
our own repo boxes, where we build the trees to our liking.

The FreeBSD svn commits are public. So, if someone wants
to use it, and select and pick to provide a stable repo,
we would all welcome the effort. If it's a sustained effort over
some time, clusteradm etc would probably add that repo
to the infrastructure. We can even call it the 'caveat'-repo.

It's just that asking the team that's barely keeping up
to do that *on top*, that will probably not work.

[...]
> From that short description it should be more or less 
> obvious if that approach is better/doable when opposed to quarterly 
> branches?

There's a way to find out: Try it.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Grzegorz Junka]

On 20/04/2017 17:11, Kurt Jaeger wrote:

Hi!


Fine, but would that be a good approach? Doesn't it look more like a
process change than a code change?

For me, it does not look like a process-change only.

I haven't thought through all the details, I'm going with my
intuition here (because thinking it through takes a long time).

One number: I made approx. 40 commits to quarterly trees in 2 years,
and broke it one least once, probably more often. Go to

https://secure.freshbsd.org/search?committer=pi

and then limit the view to the 8 quarterlies and check those commits.
It might as well be my sloppiness, but...


Surely, some code would need to be
changed but then again, wouldn't that be mostly configuration?

My gut feeling says it's more than a little change and a bit
of configuration.



I understand that the main problem with quarterly branches is that they 
start from an unstable edge and mature with time, then after three 
months at the most mature point they are being deleted and replaced with 
a new unstable edge. So, there is no good point of reference to use as a 
source of packages.


What I described is taking the good points (maturing the code through 
progressing version upgrades from CURRENT, through STABLE to RELEASE) 
while keeping good builds as reference points (monthly in CURRENT since 
it changes more often and partial builds may be too often for the server 
to handle, fortnightly in STABLE, and weekly in RELEASE since it is 
expected to contain least breaking changes and full builds are preferred 
over partial builds). Only X last builds would be kept for each of these 
three branches. From that short description it should be more or less 
obvious if that approach is better/doable when opposed to quarterly 
branches?


Grzegorz
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Baptiste Daroussin]

On Thu, Apr 20, 2017 at 02:13:52PM -0400, qjail1 wrote:
> I maintain a port and I have users complaining that the pkg system takes
> many months before the updated version of my port shows up in the pkg
> system.
> 
> My response is I tell them to change a line in their
> /etc/pkg/FreeBSD.conf file
> from url: "pkg+http://pkg.Freebsd.org/${ABI}/quarterly;,
> to   url: "pkg+http://pkg.Freebsd.org/${ABI}/latest;,

Tell them not to modify that file but to override it:
cat /usr/local/etc/pkg/repos/whatever.conf
FreeBSD: {
url: "pkg+http://pkg.Freebsd.org/${ABI}/latest;
}

:)

Best regards,
Bapt


signature.asc
Description: PGP signature

Re: Is pkg quarterly really needed?

[Kurt Jaeger]

Hi!

> Fine, but would that be a good approach? Doesn't it look more like a 
> process change than a code change?

For me, it does not look like a process-change only.

I haven't thought through all the details, I'm going with my
intuition here (because thinking it through takes a long time).

One number: I made approx. 40 commits to quarterly trees in 2 years,
and broke it one least once, probably more often. Go to

https://secure.freshbsd.org/search?committer=pi

and then limit the view to the 8 quarterlies and check those commits.
It might as well be my sloppiness, but...

> Surely, some code would need to be 
> changed but then again, wouldn't that be mostly configuration?

My gut feeling says it's more than a little change and a bit
of configuration.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Grzegorz Junka]

Fine, but would that be a good approach? Doesn't it look more like a 
process change than a code change? Surely, some code would need to be 
changed but then again, wouldn't that be mostly configuration?


Grzegorz


On 20/04/2017 08:44, Kurt Jaeger wrote:

Hi!


I am not sure if this is a rant in favour or against quarterly branches.
And this discussion comes up again and again quite regularly. I wonder
why ports don't follow the development model of the FreeBSD kernel?

- lack of developer time
   We have bapt who develops pkg. bdrewery, who does poudriere.
   A small group works on the ports framework.
   There are a few who report issues and fixes.
   I think that's it, and all work on huge workloads.
   They add features that are even more important than
   perfecting quarterly. Quarterly was not meant to fix all issues,
   it was meant as a test to learn what comes up if one provides
   some more stable pkg tree besides the HEAD.

- lack of maintainer and committer time
   maintainers and committers have to track lots of changes,
   and it's already hard to keep up with HEAD and quarterly.
   So many changes are never merged to quarterly, because
   it's difficult to grasp side effects.

About the 'lack-of-time': Please visit

https://bugs.freebsd.org/bugzilla/page.cgi?id=dashboard.html

and look at the numbers. Do it from time to time. Plot
the trajectory 8-} Submit patches to the bugzilla project that allows
us to track the trajectory 8-}

So, in general: trust the folks who do the complicated work, and
please react in a friendly way to issues you encounter. Report
them using bugzilla.freebsd.org. Search on bugzilla for
similar reports and add to them with additional tests,
reports etc.

If, after all this 'keeping-up' leaves you with spare brain cycles,
start submitting patches, and learn the big picture. It's amazingly
complex!


Then it would be a matter of creating a scheme for url addresses for
easy access to these folders with build packages.

The scheme has to be implemented in the tools.



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Baho Utot]

On 04/20/17 07:29, Mathieu Arnold wrote:

Le 20/04/2017 à 13:04, Julian Elischer a écrit :

On 20/4/17 5:15 pm, Mathieu Arnold wrote:

Le 20/04/2017 à 10:49, Torfinn Ingolfsen a écrit :

On Thu, Apr 20, 2017 at 8:00 AM, Kurt Jaeger  wrote:

Hi!


On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer
 wrote:

quarterly however is broken because the pkg mirors discard it all
at the
time of update.

Do they have to?
Why couldn't pkg mirrors keep say, the four latest quarterly sets
all the time?

Because the URL for the latest quarterly is one stable URL.

Obviously this has to be changed. As I wrote:
"No extra work involved once the setup is configured and tested".
So yes, there is some work needed, but it would be a one-time job.

If anyone has any real arguments as to why the FreeBSD project can't
do it this way, let's hear them.
HTH


I am not exactly sure what you are asking for, to keep the previous, not
updated, quarterly package repositories ? say, in latest-1 latest-2
latest-3... ?


What purpose would that serve ? I mean, they would not be updated.


exactly! that's what is often needed...  something that is not updated..


I still do not understand, if you need something that is not updated,
then do not update...



/ rant on

I tried that when I first transisioned from linux to FreeBSD.  I had one 
BITCH of a time simply because I could not find one set of base/ports 
that would give me a stable desktop.  It took me 6 months of random 
reboots ( the pc would just reboot for no reason, even if it was doing 
nothing ) packages that were in compatible with each other ( version 
hell ).  I finally got my desktop settle down,  then I became stupid and 
updated it only to go thru the process all over again.  I started using 
synth ( now I can not use it because of the last dust up that occurred, 
I don't trust it now ),  and now I have nothing to trust to get me out 
of trouble. I can not go back to a know version of base/ports that will 
together as I have not found the combination.


Now before you think I am only complaining here is a little bit of 
history. I came from Arch then moved to LFS because of the rolling 
release.  I know what a terrible time you have to keep things updated 
and working together.  The exception with LFS you can go back to a known 
point ( where things are known to work together ) and restart from there 
if you have an entire mess and must start over.


I am currently thinking of returning to LFS simply because updating 
FreeBSD and ports makes my asshole pucker.  After updating will I be 
left with something that works or will I be cussing myself for being 
stupid and updating, when I sould have left it well enough alone.


I really wanted for FreeBSD to work out so I could have one system for 
my servers, desktop and raspberry pi computers.


/rant off

Sorry if I offended anyone, I just wanted to share my delving in FreeBSD

Well carry on and have fun my friends.

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Michelle Sullivan]

Miroslav Lachman wrote:


It is not just about updates but about new installs too - if you have 
dozens of machines for customers and you need them all in the same 
version. Then some customer need some package not installed on his 
machine and you cannot run "pkg install somepackage" because then you 
will end up with upgrade of already installed packages (dependencies) 
before new package from current quaterly branch is installed.


(I do not use this scheme, but I understand the environment where 
somebody needs frozen pkg repo for much longer time than 3 months)

Create your own snapshot... it has 2 immediate and distinct advantages:

1/ Its frozen across all your systems (which means when random updates 
on how the ports system works are applied breaking everything you're not 
affected.)
2/ You can apply patches (security patches) as you need them instead of 
having to upgrade/re-snapshot because the port manager refuses/ignores 
requests to update the previous snapshot (the one you settled on)..


Then when all that is done, you can do as I did, fork the entire lot 
into a secure, up to date and working tree/build system (for OS and 
ports) where you actually have a working and reliable production system 
rather than a moving target... then you can remove all the bloat and 
unnecessary crap from the base OS and replace it all with ports stuff so 
that the base OS doesn't need upgrading unless there is a 
libc/kernel/etc security issue...  Oh wait - that's exactly what I did 
as well... you get the idea..  don't argue for it, just do it yourself 
its a lot less of a waste of energy and you get exactly what you want/need.


Regards,

--
Michelle Sullivan
http://www.mhix.org/

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Miroslav Lachman]

Mathieu Arnold wrote on 2017/04/20 13:29:

Le 20/04/2017 à 13:04, Julian Elischer a écrit :

On 20/4/17 5:15 pm, Mathieu Arnold wrote:

Le 20/04/2017 à 10:49, Torfinn Ingolfsen a écrit :



I am not exactly sure what you are asking for, to keep the previous, not
updated, quarterly package repositories ? say, in latest-1 latest-2
latest-3... ?


What purpose would that serve ? I mean, they would not be updated.


exactly! that's what is often needed...  something that is not updated..


I still do not understand, if you need something that is not updated,
then do not update...


It is not just about updates but about new installs too - if you have 
dozens of machines for customers and you need them all in the same 
version. Then some customer need some package not installed on his 
machine and you cannot run "pkg install somepackage" because then you 
will end up with upgrade of already installed packages (dependencies) 
before new package from current quaterly branch is installed.


(I do not use this scheme, but I understand the environment where 
somebody needs frozen pkg repo for much longer time than 3 months)


Miroslav Lachman
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Mathieu Arnold]

Le 20/04/2017 à 13:04, Julian Elischer a écrit :
> On 20/4/17 5:15 pm, Mathieu Arnold wrote:
>> Le 20/04/2017 à 10:49, Torfinn Ingolfsen a écrit :
>>> On Thu, Apr 20, 2017 at 8:00 AM, Kurt Jaeger  wrote:
 Hi!

> On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer
>  wrote:
>> quarterly however is broken because the pkg mirors discard it all
>> at the
>> time of update.
> Do they have to?
> Why couldn't pkg mirrors keep say, the four latest quarterly sets
> all the time?
 Because the URL for the latest quarterly is one stable URL.
>>> Obviously this has to be changed. As I wrote:
>>> "No extra work involved once the setup is configured and tested".
>>> So yes, there is some work needed, but it would be a one-time job.
>>>
>>> If anyone has any real arguments as to why the FreeBSD project can't
>>> do it this way, let's hear them.
>>> HTH
>>
>> I am not exactly sure what you are asking for, to keep the previous, not
>> updated, quarterly package repositories ? say, in latest-1 latest-2
>> latest-3... ?
>>
>>
>> What purpose would that serve ? I mean, they would not be updated.
>
> exactly! that's what is often needed...  something that is not updated..

I still do not understand, if you need something that is not updated,
then do not update...

-- 
Mathieu Arnold

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Julian Elischer]

On 20/4/17 5:18 pm, Mathieu Arnold wrote:

Le 20/04/2017 à 11:15, Mathieu Arnold a écrit :

Le 20/04/2017 à 10:49, Torfinn Ingolfsen a écrit :

On Thu, Apr 20, 2017 at 8:00 AM, Kurt Jaeger  wrote:

Hi!


On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  wrote:

quarterly however is broken because the pkg mirors discard it all at the
time of update.

Do they have to?
Why couldn't pkg mirrors keep say, the four latest quarterly sets
all the time?

Because the URL for the latest quarterly is one stable URL.

Obviously this has to be changed. As I wrote:
"No extra work involved once the setup is configured and tested".
So yes, there is some work needed, but it would be a one-time job.

If anyone has any real arguments as to why the FreeBSD project can't
do it this way, let's hear them.
HTH

I am not exactly sure what you are asking for, to keep the previous, not
updated, quarterly package repositories ? say, in latest-1 latest-2
latest-3... ?

So, that should be quarterly-1, quarterly-2..., not latest :-)

2017-Q1, 2017-Q2, 2017-Q3   etc.




What purpose would that serve ? I mean, they would not be updated.




___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Julian Elischer]

On 20/4/17 5:15 pm, Mathieu Arnold wrote:

Le 20/04/2017 à 10:49, Torfinn Ingolfsen a écrit :

On Thu, Apr 20, 2017 at 8:00 AM, Kurt Jaeger  wrote:

Hi!


On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  wrote:

quarterly however is broken because the pkg mirors discard it all at the
time of update.

Do they have to?
Why couldn't pkg mirrors keep say, the four latest quarterly sets
all the time?

Because the URL for the latest quarterly is one stable URL.

Obviously this has to be changed. As I wrote:
"No extra work involved once the setup is configured and tested".
So yes, there is some work needed, but it would be a one-time job.

If anyone has any real arguments as to why the FreeBSD project can't
do it this way, let's hear them.
HTH


I am not exactly sure what you are asking for, to keep the previous, not
updated, quarterly package repositories ? say, in latest-1 latest-2
latest-3... ?


What purpose would that serve ? I mean, they would not be updated.


exactly! that's what is often needed...  something that is not updated..
having spent 20 years embedding FreeBSD into products I can tell you 
that the commercial reality is a requirement for FROZEN (or near to 
it) contents. not contents changing all the time. ports/pkgs that need 
to be updated for security reasons are the exception, not the rule. 
And since handling a security issue usually required extra paperwork 
anyhow, they are usually handled in a different workflow.


here's a workflow I'd find less annoying.. or something similar:

quarterly snapshot into .../quarterly/$DATE/initial/All/
   this never changes again. it is a snapshot of the head set of 
patches at that time. no recomipiles needed.

an additional directory: .../quarterly/$DATE/updating/All...
all files in 'updating' start out as symlinks to 
../../initial/All/$filename and are replaced if the files get updated.

   ( or if name changes.. maybe we get a choice of two)
For the people who want to leap from quarterly to quarterly we have a
symlink from "quarterly/Latest" to $DATE/updating/ and  from 
quarterly/Initial to $DATE/initial


Each quarter  a new set is created with new dates, and the symlinks 
are recreated.
also create in each real directory, a file that contains the (possibly 
relative) URL so we know what to point our pkg.cfg at if we don't want 
to follow the rolling version.


how long you leave the old ones there is up to you, but someone 
mirroring might decide to leave them for ever if they had the space.

but at least leave them for a whole quarter...










___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Mathieu Arnold]

Le 20/04/2017 à 11:15, Mathieu Arnold a écrit :
> Le 20/04/2017 à 10:49, Torfinn Ingolfsen a écrit :
>> On Thu, Apr 20, 2017 at 8:00 AM, Kurt Jaeger  wrote:
>>> Hi!
>>>
 On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  
 wrote:
> quarterly however is broken because the pkg mirors discard it all at the
> time of update.
 Do they have to?
 Why couldn't pkg mirrors keep say, the four latest quarterly sets
 all the time?
>>> Because the URL for the latest quarterly is one stable URL.
>> Obviously this has to be changed. As I wrote:
>> "No extra work involved once the setup is configured and tested".
>> So yes, there is some work needed, but it would be a one-time job.
>>
>> If anyone has any real arguments as to why the FreeBSD project can't
>> do it this way, let's hear them.
>> HTH
>
> I am not exactly sure what you are asking for, to keep the previous, not
> updated, quarterly package repositories ? say, in latest-1 latest-2
> latest-3... ?

So, that should be quarterly-1, quarterly-2..., not latest :-)

>
> What purpose would that serve ? I mean, they would not be updated.
>
>

-- 
Mathieu Arnold




signature.asc
Description: OpenPGP digital signature

Re: Is pkg quarterly really needed?

[Mathieu Arnold]

Le 20/04/2017 à 10:49, Torfinn Ingolfsen a écrit :
> On Thu, Apr 20, 2017 at 8:00 AM, Kurt Jaeger  wrote:
>> Hi!
>>
>>> On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  wrote:
 quarterly however is broken because the pkg mirors discard it all at the
 time of update.
>>> Do they have to?
>>> Why couldn't pkg mirrors keep say, the four latest quarterly sets
>>> all the time?
>> Because the URL for the latest quarterly is one stable URL.
> Obviously this has to be changed. As I wrote:
> "No extra work involved once the setup is configured and tested".
> So yes, there is some work needed, but it would be a one-time job.
>
> If anyone has any real arguments as to why the FreeBSD project can't
> do it this way, let's hear them.
> HTH


I am not exactly sure what you are asking for, to keep the previous, not
updated, quarterly package repositories ? say, in latest-1 latest-2
latest-3... ?


What purpose would that serve ? I mean, they would not be updated.


-- 
Mathieu Arnold




signature.asc
Description: OpenPGP digital signature

Re: Is pkg quarterly really needed?

[Kurt Jaeger]

Hi!

> >> On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  
> >> wrote:
> >> > quarterly however is broken because the pkg mirors discard it all at the
> >> > time of update.

> >> Do they have to?
> >> Why couldn't pkg mirrors keep say, the four latest quarterly sets
> >> all the time?

> > Because the URL for the latest quarterly is one stable URL.

> Obviously this has to be changed. As I wrote:
> "No extra work involved once the setup is configured and tested".
> So yes, there is some work needed, but it would be a one-time job.

> If anyone has any real arguments as to why the FreeBSD project can't
> do it this way, let's hear them.

Lack of developer- and clusteradm-time. See my other posting.
Time deficit goes directly to the core of pkg/ports stuff, which
is, due to the complexity, limited to very few very skilled people
that happen to find time to solve the problem.

It's not a trivial problem, you can look at almost all source
code ecosystems (OSX, Microsoft, Solaris 8-}, debian, etc):
All struggle with the rate of change and the complexity of the task.

So it's not helping to yell louder at the folks working on it,
so please yell smarter 8-} Use a yelling tool like bugzilla,
and fine-tune your yelling by providing test cases and patches.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Torfinn Ingolfsen]

On Thu, Apr 20, 2017 at 8:00 AM, Kurt Jaeger  wrote:
> Hi!
>
>> On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  wrote:
>> > quarterly however is broken because the pkg mirors discard it all at the
>> > time of update.
>
>> Do they have to?
>> Why couldn't pkg mirrors keep say, the four latest quarterly sets
>> all the time?
>
> Because the URL for the latest quarterly is one stable URL.

Obviously this has to be changed. As I wrote:
"No extra work involved once the setup is configured and tested".
So yes, there is some work needed, but it would be a one-time job.

If anyone has any real arguments as to why the FreeBSD project can't
do it this way, let's hear them.
HTH
-- 
Regards,
Torfinn Ingolfsen
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Kurt Jaeger]

Hi!

> I am not sure if this is a rant in favour or against quarterly branches. 
> And this discussion comes up again and again quite regularly. I wonder 
> why ports don't follow the development model of the FreeBSD kernel?

- lack of developer time
  We have bapt who develops pkg. bdrewery, who does poudriere.
  A small group works on the ports framework.
  There are a few who report issues and fixes.
  I think that's it, and all work on huge workloads.
  They add features that are even more important than
  perfecting quarterly. Quarterly was not meant to fix all issues,
  it was meant as a test to learn what comes up if one provides
  some more stable pkg tree besides the HEAD.

- lack of maintainer and committer time
  maintainers and committers have to track lots of changes,
  and it's already hard to keep up with HEAD and quarterly.
  So many changes are never merged to quarterly, because
  it's difficult to grasp side effects.

About the 'lack-of-time': Please visit

https://bugs.freebsd.org/bugzilla/page.cgi?id=dashboard.html

and look at the numbers. Do it from time to time. Plot
the trajectory 8-} Submit patches to the bugzilla project that allows
us to track the trajectory 8-}

So, in general: trust the folks who do the complicated work, and
please react in a friendly way to issues you encounter. Report
them using bugzilla.freebsd.org. Search on bugzilla for
similar reports and add to them with additional tests,
reports etc.

If, after all this 'keeping-up' leaves you with spare brain cycles,
start submitting patches, and learn the big picture. It's amazingly
complex!

> Then it would be a matter of creating a scheme for url addresses for 
> easy access to these folders with build packages.

The scheme has to be implemented in the tools.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Grzegorz Junka]

On 20/04/2017 05:37, Mark Linimon wrote:

I understand that having the quarterlies is not meeting your use case.
You've said that.  We get it.

So you want some kind of running -quarterly branch.

But where are the N hours of work per week to QA all the patches to
the -quarterly branch, or a -stable branch, or whatever people seem
to demand, to come from?

This is a serious -- if very irritated -- request.

We've moved from a "we don't have enough person-power to manage a ports
branch" to "we kinda have enough person-power to manage both head and a
kinda-branch."  OK.  That isn't meeting all the use cases.  Understood.
(...)

Honestly without some volunteers to do the _hard_, _unrewarding_, work
to QA the ports tree, this is all either a) just talk, or b) people
wanting volunteers to provide professional-level support, for free.

tl;dr: provide some resources, or don't.  I am getting to the point where
I don't care either way.  All I see is the people who are doing actual work
get poked in the eye.



I am not sure if this is a rant in favour or against quarterly branches. 
And this discussion comes up again and again quite regularly. I wonder 
why ports don't follow the development model of the FreeBSD kernel? In 
short:


1. CURRENT - latest upgrades from upstreams, a broken port or a port 
that breaks other ports shouldn't be committed but not all option 
combinations are fully tested. Don't rebuild all ports, at least not 
often (maybe once a month), but rebuild all dependant ports of a port 
whenever a change in that port has been made.


2. STABLE - Only upgrade ports to the version from CURRENT when users 
haven't reported any issues with any combination of options for that 
port for some agreeable period of time since the upgrade in CURRENT 
(e.g. 2 weeks, a month). Rebuild all ports more often than in CURRENT 
(e.g. fortnight) but also not too often. Like in CURRENT rebuild all 
dependant ports whenever a port on which they depend has changed.


3. RELEASE (optional) - Like STABLE - only upgrade ports to the version 
from STABLE if no issues with any combination of options has been 
reported for some agreeable period of time since the upgrade in STABLE 
(e.g. 2-3 months this time). Rebuild all ports more often than in STABLE 
(e.g. once a week). Also like in STABLE rebuild all dependant ports when 
a port on which they depend changes.


Each branch would keep X latest full rebuilds (one, two, three - 
depending on resource availability) and the partial rebuilds in between 
them when dependant ports are rebuild - I think poudriere would copy 
them to the latest directory with packages by default.


This could give something folders like:

CURRENT month 1
any partial rebuilds on top of month 1
CURRENT month 2
any partial rebuilds on top of month 2
...

STABLE week 1
any partial rebuilds on top of week 1
STABLE week 3
any partial rebuild on top of week 3

RELEASE week 1
any partial rebuilds on top of week 1
RELEASE week 2
any partial rebuilds on top of week 2
RELEASE week 3
any partial rebuilds on top of week 3

etc.

Then it would be a matter of creating a scheme for url addresses for 
easy access to these folders with build packages.


Grzegorz

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Michael Gmelin]

On Thu, 20 Apr 2017 00:37:22 -0500
Mark Linimon  wrote:

> I understand that having the quarterlies is not meeting your use case.
> You've said that.  We get it.
> 
> So you want some kind of running -quarterly branch.
> 
> But where are the N hours of work per week to QA all the patches to
> the -quarterly branch, or a -stable branch, or whatever people seem
> to demand, to come from?
> 
> This is a serious -- if very irritated -- request.
> 
> We've moved from a "we don't have enough person-power to manage a
> ports branch" to "we kinda have enough person-power to manage both
> head and a kinda-branch."  OK.  That isn't meeting all the use
> cases.  Understood.
> 
> Are you going to volunteer for a team to run that QA?  Who else do you
> think should be on it?  Clearly the current volunteers don't have the
> bandwidth.  It is hard enough just to kep ports-head building.  Where
> do the hours come from?
> 
> You're comparing your expectations of the output of what a
> professional QA team would do, to the work that N volunteers do.
> Obviously the results are not comparable.  It's crazy to think that
> they would be.
> 
> Honestly without some volunteers to do the _hard_, _unrewarding_, work
> to QA the ports tree, this is all either a) just talk, or b) people
> wanting volunteers to provide professional-level support, for free.
> 
> tl;dr: provide some resources, or don't.  I am getting to the point
> where I don't care either way.  All I see is the people who are doing
> actual work get poked in the eye.
> 

Answering one email in the thread to provide feedback on my experience.

After some time it took to adapt, I find quarterly to be extremely
useful to me, because

  a) as a maintainer, it provides a natural deadline when
 updates should be in the ports tree (as many users will use that
 for the next three months)
  b) it's the first time I'm actually using binaries from project
 servers on a few private hosts and vms
  c) as professional users, running our own poudriere builders,
 quarterly branches are useful as a baseline for our ports tree and
 patches to it. As many things in business are done on a quarterly
 basis, we simply create a new builder every quarter, build our
 package set, test the upgrades on staging machines and then change
 the repo URL on all productions servers and upgrade.

So, even though things might not be perfect, to me it's a great
improvement compared to the previous situation and I'm grateful to those
who put lots of effort into it.

-m

-- 
Michael Gmelin
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Ethan Grammatikidis]

On Thu, Apr 20, 2017, at 06:22 AM, Mark Linimon wrote:
> On Wed, Apr 19, 2017 at 04:37:05PM -0400, scratch65...@att.net wrote:
> > (Right now, it's quite hard to resist the paranoid suspicion that
> > maybe this crazy, anti-real-user behavior is a subtle way to kill
> > freebsd altogether by driving away the non-hobbyists.)
> 
> That's one explanation.
> 
> The other, possible, explanation, is that the efforts of a group of
> volunteers isn't adequate enough for every use case -- including your own.

This!!! I used to help out a little with a Linux distro, many years ago when it 
was feasible for a handful of hobyists to do so in their spare time. The 
experience was sobering. In about 5 years I went from building my own LFS 
system without any help (I wasn't on the net) to seeing a bunch of people just 
barely keep up as Xorg and Gtk+ grew circular dependency chains, some users 
demanded stability while others wanted updates, and of course compile times 
were rapidly increasing.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Ethan Grammatikidis]

On Tue, Apr 18, 2017, at 02:54 PM, qjail1 wrote:
> I maintain a port and I have users complaining that the pkg system takes 
> many months before the updated version of my port shows up in the pkg 
> system.
> 
> My response is I tell them to change a line in their 
> /etc/pkg/FreeBSD.conf file
> from url: "pkg+http://pkg.Freebsd.org/${ABI}/quarterly;,
> to   url: "pkg+http://pkg.Freebsd.org/${ABI}/latest;,
> 
> The old pkg system never had this quarterly update cycle and I see no 
> reason to have it now when its so easy to over ride the default.
> 
> Why not just change the default to "latest" and save on all the overhead 
> of the quarterly cycle?

I have to say that if pkg abandoned the cautious update cycle, I'd abandon 
FreeBSD. I've really had enough of constant upgrades. The one thing I have 
which needs constant upgrades is a Flash video downloader. I keep that in ~/bin 
and use its built-in upgrade mechanism when required, which is simple. I don't 
need to get it via some root-only mechanism. It doesn't have a man page of 
course, but --help|less is easy enough.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Kurt Jaeger]

Hi!

> On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  wrote:
> > quarterly however is broken because the pkg mirors discard it all at the
> > time of update.

> Do they have to?
> Why couldn't pkg mirrors keep say, the four latest quarterly sets
> all the time?

Because the URL for the latest quarterly is one stable URL.

If an enduser wants to access a particular quarterly, the enduser
needs to edit the repo URL. Which probably kills approx. 95%
of the use-cases.

One could do some process where every quarterly repo communicates
it's quarterly URL back to the client accessing it and pkg automagically
adapts the repo URL, but that does sound fragile, too.

-- 
p...@opsec.eu+49 171 3101372 3 years to go !
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Torfinn Ingolfsen]

On Thu, Apr 20, 2017 at 1:30 AM, Julian Elischer  wrote:
> quarterly however is broken because the pkg mirors discard it all at the
> time of update.
>

Do they have to?
Why couldn't pkg mirrors keep say, the four latest quarterly sets all the time?
This would increase the usability of quarterly packages, at users own
risk, with only more diskspace as the expense for the FreeBSD projects
/ pkg mirrors.
No extra work involved once the setup is configured and tested.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Mark Linimon]

I understand that having the quarterlies is not meeting your use case.
You've said that.  We get it.

So you want some kind of running -quarterly branch.

But where are the N hours of work per week to QA all the patches to
the -quarterly branch, or a -stable branch, or whatever people seem
to demand, to come from?

This is a serious -- if very irritated -- request.

We've moved from a "we don't have enough person-power to manage a ports
branch" to "we kinda have enough person-power to manage both head and a
kinda-branch."  OK.  That isn't meeting all the use cases.  Understood.

Are you going to volunteer for a team to run that QA?  Who else do you
think should be on it?  Clearly the current volunteers don't have the
bandwidth.  It is hard enough just to kep ports-head building.  Where
do the hours come from?

You're comparing your expectations of the output of what a professional
QA team would do, to the work that N volunteers do.  Obviously the results
are not comparable.  It's crazy to think that they would be.

Honestly without some volunteers to do the _hard_, _unrewarding_, work
to QA the ports tree, this is all either a) just talk, or b) people
wanting volunteers to provide professional-level support, for free.

tl;dr: provide some resources, or don't.  I am getting to the point where
I don't care either way.  All I see is the people who are doing actual work
get poked in the eye.

mcl
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Mark Linimon]

On Wed, Apr 19, 2017 at 04:37:05PM -0400, scratch65...@att.net wrote:
> (Right now, it's quite hard to resist the paranoid suspicion that
> maybe this crazy, anti-real-user behavior is a subtle way to kill
> freebsd altogether by driving away the non-hobbyists.)

That's one explanation.

The other, possible, explanation, is that the efforts of a group of
volunteers isn't adequate enough for every use case -- including your own.

But, of course, feel free to cast aspersions wherever and whenever.  We're
just machines, we have no feelings whatsoever.

Now if no one minds, I'm going to go back to contemplate the existential
question of "why do I even bother trying to fix things".

mcl
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Julian Elischer]

On 20/4/17 6:29 am, Dewayne Geraghty wrote:

Scratch65535, I think your best solution is to use latest and upgrade when
you need to.  Unlike Freddie's comment re only desktop users using latest.
I ONLY upgrade my local svn of ports when there's a vulnerability or
significant (for users) functional improvement of a port.

It is a labour intensive exercise, monitoring CVE's for all
externally-facing applications.

Its a nice idea having a snapshot of ports, from the perspective of
consistency, but that model doesnt suite our risk appetite on multiple
levels; and in our view back-porting fixes to a quarterly snapshot - a good
idea from a security perspective it is a really bad idea from a
consistency/administrative/audit perspective.


We mirror the ports tree (and base) into p4 and also as svn, and use 
this to check out the head branch to whatever release we need.
Our scripts are capable of checking out a particular port at a 
(slightly) different rev to the default rev used for the rest, as 
sometimes we find we need a slightly newer rev of one port or 
another.  This sometimes doesn't work if there are framework changes 
that affect the port but mostly we find that it's ok if you just want 
to bump a port up a small amount to catch a bugfix,or take it back a 
bit to avoid a regression. We also do sparse checkouts of the ports 
tree ot save time, but that's another issue..


We therefore have all out pkgs (which we store with each release) at 
the same level of source tree so they all match.




How the ports infrastructure can meet many conflicting objectives is
something that we (the consumers of the ports service) must decide for our
circumstance.  The use-the-latest paradigm suits individuals that manage
their individual machine, but when you manage multiple clients' servers,
the requirements are different (try meeting a SAS70-II/SAE16-SOC2, ISO27001
SOA, NIST 800-53r5, etc)

On a non-audit level, Microsoft might hold to monthly updates/fixes ("patch
Tuesday") but bad guys don't.
Regards, Dewayne.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Julian Elischer]

On 20/4/17 4:37 am, scratch65...@att.net wrote:

[Default] On Tue, 18 Apr 2017 15:57:02 +0100, krad
 wrote:


quarterly does seem very cautious, maybe a monthly might be a good
alternative.

I have to STRONGLY disagree.

Right now, pkg isn't smart enough not to use version-skewed bits.
Which means that, for those of us trying to use freebsd and the
ports/pkgs as *tools* wherewith to do other work, we have to run
pkg upgrade -fy every cursëd quarter, not just when the minor
number changes.  With crappy access to the inet (something true
of most of the USA and Canada outside Really Big Cities) that
costs half a day or more.

If it were every month, I'd with unconsolable regret abandon
freebsd for linux (ptui!).  (Right now, it's quite hard to resist
the paranoid suspicion that maybe this crazy, anti-real-user
behavior is a subtle way to kill freebsd altogether by driving
away the non-hobbyists.)

I have to agree with you. It is most frustrating
The only way to sanity is to totally ignore the quarterly releases, 
and if you have changes, use poudriere to build what you need yourself 
once a month or whatever, or if you don't have changes, take snapshots 
of the entire pkg tree every now and then. (we do both for different 
reasons).. My snapshot is kept out on an amazon machine we have, as 
it's purpose is to "freeze in time" the head branch of the pkg tree. 
This means we can come back in a week and get  a new pkg that we now 
need and know it is compatible with the other ones we have. I don't 
have to download the entire collection through my crappy link. just 
the ones I need.
The trouble is that the people doing ports and pkg management really 
don't really have production systems in mind and rarely really 
understand the requirements of such. (reproducible builds from 
scratch, and the ability to append to an existing  frozen-in-time 
release (for debug or bug-fix reasons). They ESPECIALLY don't 
understand the requirement to have access to old sets of packages, so 
you need to do it yourself.

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"




___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Julian Elischer]

On 19/4/17 12:13 am, Freddie Cash wrote:

On Tue, Apr 18, 2017 at 7:57 AM, krad  wrote:


quarterly does seem very cautious, maybe a monthly might be a good
alternative. I can understand people being hesitant about latest though. I
guess these are not the people who ask though. Maybe the real answer though
is to have a specific repo for that port for the bleeding edge people  much
like launchpad on ubuntu. It might get complicated though for big
dependency trees though.


​latest/ is good for desktop users who only care about running the very
latest versions of everything.

quarterly/ is good for server users who want to make sure that installing a
new piece of software won't require an upgrade of all the currently
installed software (repo hasn't changed since the last install).  And for
desktop users who prefer to use their computers for doing work instead of
spending all their time chasing after the "latest" flashy bling bling.  :)

quarterly/ also gets security fixes back-ported into it (on a
per-maintainer basis so it's not 100% coverage yet), so you can stay secure
without chasing new software versions.

IOW, don't change the infrastructure, it's working nicely.  Just educate
the users.  :D


quarterly however is broken because the pkg mirors discard it all at the time 
of update.
meaning you can not come back later to get one you forgot..
and sometimes you can get half way through, and everything breaks becuase you 
happened to select  teh wrong date to do your work.





On 18 April 2017 at 14:54, qjail1  wrote:


I maintain a port and I have users complaining that the pkg system takes
many months before the updated version of my port shows up in the pkg
system.

My response is I tell them to change a line in their

/etc/pkg/FreeBSD.conf

file
from url: "pkg+http://pkg.Freebsd.org/${ABI}/quarterly;,
to   url: "pkg+http://pkg.Freebsd.org/${ABI}/latest;,

The old pkg system never had this quarterly update cycle and I see no
reason to have it now when its so easy to over ride the default.

Why not just change the default to "latest" and save on all the overhead
of the quarterly cycle?
___
freebsd-questi...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe
@freebsd.org"


___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"






___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Dewayne Geraghty]

Scratch65535, I think your best solution is to use latest and upgrade when
you need to.  Unlike Freddie's comment re only desktop users using latest.
I ONLY upgrade my local svn of ports when there's a vulnerability or
significant (for users) functional improvement of a port.

It is a labour intensive exercise, monitoring CVE's for all
externally-facing applications.

Its a nice idea having a snapshot of ports, from the perspective of
consistency, but that model doesnt suite our risk appetite on multiple
levels; and in our view back-porting fixes to a quarterly snapshot - a good
idea from a security perspective it is a really bad idea from a
consistency/administrative/audit perspective.

How the ports infrastructure can meet many conflicting objectives is
something that we (the consumers of the ports service) must decide for our
circumstance.  The use-the-latest paradigm suits individuals that manage
their individual machine, but when you manage multiple clients' servers,
the requirements are different (try meeting a SAS70-II/SAE16-SOC2, ISO27001
SOA, NIST 800-53r5, etc)

On a non-audit level, Microsoft might hold to monthly updates/fixes ("patch
Tuesday") but bad guys don't.
Regards, Dewayne.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[scratch65535]

[Default] On Tue, 18 Apr 2017 15:57:02 +0100, krad
 wrote:

>quarterly does seem very cautious, maybe a monthly might be a good
>alternative. 

I have to STRONGLY disagree.  

Right now, pkg isn't smart enough not to use version-skewed bits.
Which means that, for those of us trying to use freebsd and the
ports/pkgs as *tools* wherewith to do other work, we have to run
pkg upgrade -fy every cursëd quarter, not just when the minor
number changes.  With crappy access to the inet (something true
of most of the USA and Canada outside Really Big Cities) that
costs half a day or more.  

If it were every month, I'd with unconsolable regret abandon
freebsd for linux (ptui!).  (Right now, it's quite hard to resist
the paranoid suspicion that maybe this crazy, anti-real-user
behavior is a subtle way to kill freebsd altogether by driving
away the non-hobbyists.)
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[qjail1]

Jan Beich wrote:

qjail1  writes:


I maintain a port and I have users complaining that the pkg system
takes many months before the updated version of my port shows up in
the pkg system.


Better ask committer assigned to your bug to add MFH tag or send an
email to ports-secteam@ (and CC portmgr@) which commit to backport.
For leaf ports such requests are unlikely to be declined, just keep
in mind risks due to using old dependencies and possible regressions.



My port is nothing but two sh scripts and an example directory plus the 
man pages. It has no dependencies and no regressions. Does this port 
qualify for MFH tag? Is this something I can put in the port Makefile?

___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Jan Beich]

qjail1  writes:

> I maintain a port and I have users complaining that the pkg system
> takes many months before the updated version of my port shows up in
> the pkg system.

Better ask committer assigned to your bug to add MFH tag or send an
email to ports-secteam@ (and CC portmgr@) which commit to backport.
For leaf ports such requests are unlikely to be declined, just keep
in mind risks due to using old dependencies and possible regressions.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[Freddie Cash]

On Tue, Apr 18, 2017 at 7:57 AM, krad  wrote:

> quarterly does seem very cautious, maybe a monthly might be a good
> alternative. I can understand people being hesitant about latest though. I
> guess these are not the people who ask though. Maybe the real answer though
> is to have a specific repo for that port for the bleeding edge people  much
> like launchpad on ubuntu. It might get complicated though for big
> dependency trees though.
>

​latest/ is good for desktop users who only care about running the very
latest versions of everything.

quarterly/ is good for server users who want to make sure that installing a
new piece of software won't require an upgrade of all the currently
installed software (repo hasn't changed since the last install).  And for
desktop users who prefer to use their computers for doing work instead of
spending all their time chasing after the "latest" flashy bling bling.  :)

quarterly/ also gets security fixes back-ported into it (on a
per-maintainer basis so it's not 100% coverage yet), so you can stay secure
without chasing new software versions.

IOW, don't change the infrastructure, it's working nicely.  Just educate
the users.  :D



>
> On 18 April 2017 at 14:54, qjail1  wrote:
>
> > I maintain a port and I have users complaining that the pkg system takes
> > many months before the updated version of my port shows up in the pkg
> > system.
> >
> > My response is I tell them to change a line in their
> /etc/pkg/FreeBSD.conf
> > file
> > from url: "pkg+http://pkg.Freebsd.org/${ABI}/quarterly;,
> > to   url: "pkg+http://pkg.Freebsd.org/${ABI}/latest;,
> >
> > The old pkg system never had this quarterly update cycle and I see no
> > reason to have it now when its so easy to over ride the default.
> >
> > Why not just change the default to "latest" and save on all the overhead
> > of the quarterly cycle?
> > ___
> > freebsd-questi...@freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to "freebsd-questions-unsubscribe
> > @freebsd.org"
> >
> ___
> freebsd-ports@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
>



-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Is pkg quarterly really needed?

[krad]

quarterly does seem very cautious, maybe a monthly might be a good
alternative. I can understand people being hesitant about latest though. I
guess these are not the people who ask though. Maybe the real answer though
is to have a specific repo for that port for the bleeding edge people  much
like launchpad on ubuntu. It might get complicated though for big
dependency trees though.


On 18 April 2017 at 14:54, qjail1  wrote:

> I maintain a port and I have users complaining that the pkg system takes
> many months before the updated version of my port shows up in the pkg
> system.
>
> My response is I tell them to change a line in their /etc/pkg/FreeBSD.conf
> file
> from url: "pkg+http://pkg.Freebsd.org/${ABI}/quarterly;,
> to   url: "pkg+http://pkg.Freebsd.org/${ABI}/latest;,
>
> The old pkg system never had this quarterly update cycle and I see no
> reason to have it now when its so easy to over ride the default.
>
> Why not just change the default to "latest" and save on all the overhead
> of the quarterly cycle?
> ___
> freebsd-questi...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe
> @freebsd.org"
>
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"