Re: [arch-dev-public] Add active Python versions to the repos
On Thu, 2020-11-26 at 21:21 +0100, Jelle van der Waa wrote: > On 21/11/2020 15:34, Filipe Laíns via arch-dev-public wrote: > > Hi all, > > > > I want to propose adding all active Python versions to [community], not > > just the latest one. This would only entail adding the interpreter > > itself, no other packages. > > > > Having access to interpreters for older active versions is really > > helpful for Python developers. This allows them to easily run test > > suites against older versions. It is very common for developers to > > maintain software against a couple major releases. Tools like tox or > > nox are able to automate testing against multiple Python versions, just > > needing the interpreter. > > > > The current active Python releases are: > > - 3.9 > > - 3.8 > > - 3.7 > > - 3.6 > > - 2.7 > > > > The list can be found here[1]. > > > > So, I propose introducing 2 new packages: > > - python3.7 > > - python3.6 > > > > And when we update the python package to 3.9: > > - python3.8 > > I am not super in favor of providing an old version, as we strive to > provide the latest and greatest. However some things come to mind: We would still provide the latest and greatest :P > * Does the Python foundation still maintain older Python versions and do > they provide proper security updates? It depends on the version. Python 3.6 and 3.7 are in security maintenance status. I am okay limiting my proposal to interpreters which are getting security updates, which at the time being doesn't change anything regarding the versions affected. It is also not clear in the proposal, but I have mentioned in the replies. I only want the interpreters there as long as they do not give us any issues. If a interpreter starts needing older dependencies or otherwise becomes a maintenance burden, it would be removed. > * This seems to be purely for Development right? Users would use this > Python version to bootstrap a virtualenv I guess? The target is development, yes. You are correct, users, or tools, could use the interpreter to bootstrap a virtual environment using either virtualenv, which supports using external interpreters, or the builtin venv module. > > Greetings, > > Jelle > Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sun, 2020-11-22 at 11:45 -0500, Eli Schwartz via arch-dev-public wrote: > On 11/22/20 10:05 AM, Filipe Laíns wrote: > > On Sat, 2020-11-21 at 20:24 -0500, Eli Schwartz via arch-dev-public wrote: > > > Your analysis is correct, it is indeed hell. I'm not sure why that is an > > > argument in favor of doing even more of it though. > > > > > > Now, if you were proposing to get rid of some of this, I could get > > > behind that. > > > > It was not an analysis, "hell" was an used here as an idiom. > > https://i.imgur.com/axJmn.gif > > You were unintentionally accurate. > > > > > even Python. > > > > > > I don't know if maybe you've missed the fact that we've spent like a > > > year now aggressively upgrading or removing leaf packages depending on > > > python2 in search of that glorious Promised Future where we can finally > > > remove the python2 package itself? > > > > I did not miss that. I absolutely want to see the python2 modules removed, > > not > > _necessarily_ the python2 interpreter. > > Now I'm starting to see part of the reason why people are having a > difference of opinions here. > > I think I probably speak for a bunch of people when I say I'd like to > see the interpreter dropped due to not being needed by any packages > *and* being an end of life interpreter. The interpreter is only useful > inasmuch as people use it to run software, so if we don't have any... > > I expect by the time we finally remove all reverse dependencies for it, > the gap between the python2 EOL and its removal from the distro will be > quite a bit biger than it currently is. > > > > And we're doing that even for a major incompatible release that most > > > people argue is actually a different language. > > > > > > I'm not sure how this suddenly became an argument to package a ton of > > > point releases that aren't even incompatible. > > > > Except they are... Python does not follow semver, only patch releases are > > supposed to be compatible. They do try to keep breakage to a minimum, but it > > definitely exists. One very obvious example is the introduction of "async" > > and > > "await" as reserved keywords. There are a bunch of other backwards > > incompatible > > changes, so yes, they are incompatible releases, to the point you could call > > them different languages. > > This is the first time I ever heard anyone describe "backwards > incompatible changes means it is now a different programming language". You literally mentioned that in your reply. Though I assume you meant more significant change than what is being discussed. If the languages semantics are different one could argue it is actually a different language. Anyway, this does not really matter for the issue being discussed. > I had backwards incompatible changes in a patchlevel bugfix in the > python 3.8.x release line causing immediate traceback and SystemExit on > a program's startup, does that mean every bugfix is now a different > programming language? > > (I mean this quite literally. Python bugfixed a function not working as > expected under certain situations. The program it broke was working > around the failure by manually doing $thing, the traceback happened when > you try to do $thing twice.) > > > > > I don't think having people opening bugs because they are > > > > deliberately using an older version of Python is a big problem. It > > > > hasn't been for any of the other languages, I don't think it will be > > > > here. > > > > I think this is an hypothetical that doesn't really materialize into > > > > reality. > > > > > > You're proposing something far beyond the scope of what we do for other > > > languages, and ignoring that for other languages we only do it if driven > > > to in desperation because another official repository package depends on > > > it. > > > > > > We don't introduce leaf packages of ancient versions of currently > > > packaged interpreters just for fun. I don't even think anyone else does > > > that either. > > > > That was not of the point of me bringing up other languages. The whole point > > of > > me bringing up other languages was to show that Andreas' concern is unlikely > > to > > materialize into reality. > > You think no one reports bugs for python-* packages that we should > provide the python2 version? I promise you, you're wrong. I've closed > those bugs, so I know they exist. That is not "users raising bugs that something doesn't work with an older version of python", and is not what I argued against. > You specifically brought up Java and Javascript, languages where each > application is forced to vendor their entire dependency list rather than > using system modules. For this reason, people don't report the kind of > bugs Andreas is concerned about for Java/nodejs... They are not necessarily forced, but whatever, just ignore that. > So I assumed you could not possibly be talking about Andreas' concern, > and I responded somewhat tangentially myself. We can, of course,
Re: [arch-dev-public] Add active Python versions to the repos
On Sun, 2020-11-22 at 09:43 -0600, Doug Newgard via arch-dev-public wrote: > On Sun, 22 Nov 2020 15:32:22 + > Filipe Laíns via arch-dev-public wrote: > > > On Sun, 2020-11-22 at 09:13 -0600, Doug Newgard via arch-dev-public wrote: > > > Why did you start the discussion if you plan on ignoring other people's > > > opinions anyway? > > > > > > Doug > > > > I am not ignoring Andreas' or Eli's opinion, I understand that they don't > > want > > these packages in the repos. What I am asking is why they don't want them > > there? > > I don't appreciate you trimming the paragraph I was replying to. I makes my > comments look completely different without context. Sorry, I only replied to the specific reply because my email client is messing up the format and I have not fixed it yet. I did not meant to take it out of context. > The paragraph specifically said that you see no reason to block anything as > long as someone is willing to maintain it. That means that other's opinions > in this disussion don't matter, unless I misunderstood? > > Doug That was not meant that way. Similarly to your reply, the context matters. The comment was meant as "If this helps people and does not impact you, why would you block it?", I said this because Eli seems hell bent on not having the packages in the repos. Sorry if it came out the wrong way. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sun, 2020-11-22 at 09:13 -0600, Doug Newgard via arch-dev-public wrote: > Why did you start the discussion if you plan on ignoring other people's > opinions anyway? > > Doug I am not ignoring Andreas' or Eli's opinion, I understand that they don't want these packages in the repos. What I am asking is why they don't want them there? I have addressed the concern Andreas raised. I am not sure if satisfactorily to him as he has not replied yet. Eli, so far, has presented me with the current status quo, stating this is not we currently do. Which was never what I was arguing, that's why I have started this thread, in hope of changing that. From his reply, the only part I see as actual reasoning against the proposed change is this: > I don't see any reason why [community] needs to grow to support the needs of exclusively Continuous Integration testsuite matrixes. Which I have addressed. Please let me know if I've missed any of his points. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sat, 2020-11-21 at 20:24 -0500, Eli Schwartz via arch-dev-public wrote:
> Your analysis is correct, it is indeed hell. I'm not sure why that is an
> argument in favor of doing even more of it though.
>
> Now, if you were proposing to get rid of some of this, I could get
> behind that.
It was not an analysis, "hell" was an used here as an idiom.
> > even Python.
>
> I don't know if maybe you've missed the fact that we've spent like a
> year now aggressively upgrading or removing leaf packages depending on
> python2 in search of that glorious Promised Future where we can finally
> remove the python2 package itself?
I did not miss that. I absolutely want to see the python2 modules removed, not
_necessarily_ the python2 interpreter.
> And we're doing that even for a major incompatible release that most
> people argue is actually a different language.
>
> I'm not sure how this suddenly became an argument to package a ton of
> point releases that aren't even incompatible.
Except they are... Python does not follow semver, only patch releases are
supposed to be compatible. They do try to keep breakage to a minimum, but it
definitely exists. One very obvious example is the introduction of "async" and
"await" as reserved keywords. There are a bunch of other backwards incompatible
changes, so yes, they are incompatible releases, to the point you could call
them different languages.
> > I don't think having people opening bugs because they are
> > deliberately using an older version of Python is a big problem. It
> > hasn't been for any of the other languages, I don't think it will be
> > here.
> > I think this is an hypothetical that doesn't really materialize into
> > reality.
>
> You're proposing something far beyond the scope of what we do for other
> languages, and ignoring that for other languages we only do it if driven
> to in desperation because another official repository package depends on it.
>
> We don't introduce leaf packages of ancient versions of currently
> packaged interpreters just for fun. I don't even think anyone else does
> that either.
That was not of the point of me bringing up other languages. The whole point of
me bringing up other languages was to show that Andreas' concern is unlikely to
materialize into reality.
> > > It's sad enough we still have python2 and gtk2 around. To have gcc9
> > > around and other duplicates is not what I want to see growing in
> > > Arch.
> >
> > What you call sad I call a bad UX. Why do we need to force users to
> > compile active releases of the Python interpreters themselves, which
> > can take a long time if they are building with optimization, or to
> > resort to pre-built repos with much lower security standards than us,
> > when there are people willing to maintain them?
> >
> > I can't understand how it's sad to help out users by not forcing them
> > to resort the sort of things I mentioned above, as long as we are not
> > struggling to do so. I like helping people, that's why I am a packager,
> > that is the opposite of sad for me, so I really can't understand this.
>
> I have... no idea what the problem is supposed to be here? I'm
> desperately trying to understand what the actual point of your proposal is.
>
> You want to package old versions of the python3 interpreter "but not
> modules", because "people would like to use it for tox".
People can use it tox or to create virtual environments directly. The main point
of the proposal is that people have use for the old interpreters and I would
like to provide them.
> Old versions of stuff that people need for who knows what weird reason
> is *the textbook reason* why the AUR exists. How long it takes to
> compile stuff in the AUR is not our problem, and we don't have a
> rationale to upload gcc-{4,5,6,7,8,9} because actual users need to use a
> lot of diverse compilers "to test that it still works on really old
> compilers".
AFAIK, the textbook reason why the AUR exists is to provide software that isn't
in the official repos. It being older versions should be completely incidental,
but not the "textbook reason".
> Likewise, if people want to test support for old versions of python3,
> they have options which don't involve uploading trash into [community].
I like how you refereed to the currently active Python releases as "trash", will
keep that in mind.
> - Do what normal people do, and build AUR packages they need
> - Ask FFY00 to provide signed repos
> - Use travis CI, which doesn't use distro packages but provides a
> diverse set of hand-compiled python versions, which is apparently some
> kind of gold standard for doing regression testing on lots of
> different python releases ;)
> - Use pyenv
>
> Apparently none of these are viable options in your mind, and the most
> Arch-like of them, option #1 "use the AUR" you believe to be problematic
> because it takes a long time to build with optimizations... even though
> you yourself said those packages don't enable
Re: [arch-dev-public] Orphaned packages from arcanis
On Sat, 2020-11-21 at 20:09 +0100, Morten Linderud via arch-dev-public wrote: pymol python-biopython Adopted. Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
Whoops, my email client messed up the formatting :/ Here's the reply without the quotes: Looking at some package stats for pyenv[1], we see a usage of 3,4%. I think that is a big enough number. pyenv does have another usage, which is managing custom Python interpreters, which is mainly used in CPython development, but I'd say that only a small percentage of the users use it uniquely for that. We can also look directly at the python36 and python37 packages reported, which users might be getting from the AUR or user repos. python37 has 1.18% [2] python36 has 1.63% [3] The main reason why the packages might be required, as I explained in the original email, is for package testing. It puts every maintainer and most contributors to projects that target Python 3.6 or 3.7 in need of an interpreter. Most open source Python project target at least a few different Python versions. These metrics are not perfect by any means, but I do think they present relevant enough data to show an existent need. [1] https://pkgstats.archlinux.de/api/packages/pyenv [2] https://pkgstats.archlinux.de/api/packages/python36 [3] https://pkgstats.archlinux.de/api/packages/python37 Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sat, 2020-11-21 at 19:11 +0100, Bartłomiej Piotrowski via arch-dev-public wrote: On 21/11/2020 18.48, Filipe Laíns via arch-dev-public wrote: > I understand that. I am not asking to put all releases of Python on > the > repos, only the active ones, which people are using. I presume you can back it up with numbers how widely 3.7 and 3.6 are used by Arch users. All I can see is that both have less than 30 votes in AUR. Even if I take into account how irrelevant AUR votes are, I assume the problem is exaggerated. I have some metrics that might be relevant. The people that are missing these interpreters generally use pyenv to make up for it, which is not an optimal solution, but the most common when the interpreters are missing. Looking at some package stats for pyenv[1], we see a usage of 3,4%. I think that is a big enough number. pyenv does have another usage, which is managing custom Python interpreters, which is mainly used in CPython development, but I'd say that only a small percentage of the users use it uniquely for that. We can also look directly at the python36 and python37 packages reported, which users might be getting from the AUR or user repos. python37 has 1.18% [2] python36 has 1.63% [3] The main reason why the packages might be required, as I explained in the original email, is for package testing. It puts every maintainer and most contributors to projects that target Python 3.6 or 3.7 in need of an interpreter. Most open source Python project target at least a few different Python versions. These metrics are not perfect by any means, but I do think they present relevant enough data to show an existent need. Why are we packaging software that is used by far less people but we can't package these Python interpreters which are being actively missed by people? Our approach of "YOLO push random new packages to repos" is something I never agreed with, for the record. Unfortunately it's a kingdom with almost no rules. BP [1] https://pkgstats.archlinux.de/api/packages/pyenv [2] https://pkgstats.archlinux.de/api/packages/python36 [3] https://pkgstats.archlinux.de/api/packages/python37 Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sat, 2020-11-21 at 11:24 -0600, Doug Newgard via arch-dev-public wrote: > On Sat, 21 Nov 2020 16:59:21 + > Filipe Laíns via arch-dev-public > wrote: > > > On Sat, 2020-11-21 at 16:58 +0100, Andreas Radke via arch-dev- > > public > > wrote: > > > Am Sat, 21 Nov 2020 14:34:24 +0000 > > > schrieb Filipe Laíns via arch-dev-public > > > : > > > > > > > > > > Does anyone have any big issue with this? What are your > > > > thoughts? > > > > > > > > [1] https://www.python.org/downloads/ > > > > > > > > Cheers, > > > > Filipe Laíns > > > > > > -1 > > > > > > Arch is yours. Whoever needs more and older releases on the > > > system - > > > just do it yourself! In the past we said "use abs and AUR - Arch > > > is > > > the base to make it your own". > > > > This argument can be used to deny adding any package to the repos. > > You > > need this library, tool, etc.? Just add it yourself. > > > > Why are we packaging software that is used by far less people but > > we > > can't package these Python interpreters which are being actively > > missed > > by people? > > > > > I don't want to see users raising bugs that something doesn't > > > work > > > with an older version of python. And I don't want to see these > > > requests > > > pop up every now and then to add even more stuff in different > > > versions. > > > > We already have multiple versions of Java, Ruby, Javascript, etc. > > hell, > > even Python. I don't think having people opening bugs because they > > are > > deliberately using an older version of Python is a big problem. It > > hasn't been for any of the other languages, I don't think it will > > be > > here. > > I think this is an hypothetical that doesn't really materialize > > into > > reality. > > > > > It's sad enough we still have python2 and gtk2 around. To have > > > gcc9 > > > around and other duplicates is not what I want to see growing in > > > Arch. > > > > What you call sad I call a bad UX. Why do we need to force users to > > compile active releases of the Python interpreters themselves, > > which > > can take a long time if they are building with optimization, or to > > resort to pre-built repos with much lower security standards than > > us, > > when there are people willing to maintain them? > > > > I can't understand how it's sad to help out users by not forcing > > them > > to resort the sort of things I mentioned above, as long as we are > > not > > struggling to do so. I like helping people, that's why I am a > > packager, > > that is the opposite of sad for me, so I really can't understand > > this. > > It's more concerning to me that you can't understand this argument > than > anything else so far. Arch keeps old things around in the repos when > they're > required by other things in the repos. It's a necessary evil, not > something to > be actively encouraged. I understand that. I am not asking to put all releases of Python on the repos, only the active ones, which people are using. > > > I don't want to see our distribution transformed into another > > > Debian. > > > > That is not what is happening. > > > > Cheers, > > Filipe Laíns > -- Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sat, 2020-11-21 at 16:58 +0100, Andreas Radke via arch-dev-public wrote: > Am Sat, 21 Nov 2020 14:34:24 + > schrieb Filipe Laíns via arch-dev-public > : > > > > Does anyone have any big issue with this? What are your thoughts? > > > > [1] https://www.python.org/downloads/ > > > > Cheers, > > Filipe Laíns > > -1 > > Arch is yours. Whoever needs more and older releases on the system - > just do it yourself! In the past we said "use abs and AUR - Arch is > the base to make it your own". This argument can be used to deny adding any package to the repos. You need this library, tool, etc.? Just add it yourself. Why are we packaging software that is used by far less people but we can't package these Python interpreters which are being actively missed by people? > I don't want to see users raising bugs that something doesn't work > with an older version of python. And I don't want to see these > requests > pop up every now and then to add even more stuff in different > versions. We already have multiple versions of Java, Ruby, Javascript, etc. hell, even Python. I don't think having people opening bugs because they are deliberately using an older version of Python is a big problem. It hasn't been for any of the other languages, I don't think it will be here. I think this is an hypothetical that doesn't really materialize into reality. > It's sad enough we still have python2 and gtk2 around. To have gcc9 > around and other duplicates is not what I want to see growing in > Arch. What you call sad I call a bad UX. Why do we need to force users to compile active releases of the Python interpreters themselves, which can take a long time if they are building with optimization, or to resort to pre-built repos with much lower security standards than us, when there are people willing to maintain them? I can't understand how it's sad to help out users by not forcing them to resort the sort of things I mentioned above, as long as we are not struggling to do so. I like helping people, that's why I am a packager, that is the opposite of sad for me, so I really can't understand this. > I don't want to see our distribution transformed into another Debian. That is not what is happening. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sat, 2020-11-21 at 16:59 +0100, Morten Linderud via arch-dev-public wrote: > On Sat, Nov 21, 2020 at 04:47:27PM +0100, David Runge wrote: > > On 2020-11-21 14:34:24 (+), Filipe Laíns via arch-dev-public > > wrote: > > > Hi all, > > > > > > I want to propose adding all active Python versions to > > > [community], not > > > just the latest one. This would only entail adding the > > > interpreter > > > itself, no other packages. > > > > > An alternative (in a per-user setup) can be to use pyenv [1]. It > > works > > reasonably well with tox etc. and I have used it in the past > > successfully to develop against several python interpreter > > versions. > > I'm personally not very excited for the idea of adding more python > interpreters. > I'm a bit concerned that we today say "no packages" but in the future > we relax a > bit and end up with python36-$pkgname, as it's the pragmatic option > as opposed > to blocking entire rebuilds or package updates. We can add a guideline blocking this. > What is the downsides of utilizing something like pyenv? There are > user > repositories providing older python interpreters as well if people > need it. > pyenv forces users to compile the Python interpreter themselves, which can take a long time with --enable-optimizations. None of the user repos available builds with optimizations, or has signed packages AFAIK. Of course they could in the future, but I think having the packages in the repos is much better in terms of security and usability. I run one of the user which provides these packages and I don't see myself fixing any of the issues I pointed out due to technical limitations. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Add active Python versions to the repos
On Sat, 2020-11-21 at 07:51 -0700, Zach Himsel wrote: > What would determine what the "default" python be? The latest? > > > -- > Zach Himsel > mailto://z...@himsel.net > http://zach.himsel.net Yep, that would stay exactly as it is right now. We would only additionally provide interpreters (and interpreters only!) for the older Python versions. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
[arch-dev-public] Add active Python versions to the repos
Hi all, I want to propose adding all active Python versions to [community], not just the latest one. This would only entail adding the interpreter itself, no other packages. Having access to interpreters for older active versions is really helpful for Python developers. This allows them to easily run test suites against older versions. It is very common for developers to maintain software against a couple major releases. Tools like tox or nox are able to automate testing against multiple Python versions, just needing the interpreter. The current active Python releases are: - 3.9 - 3.8 - 3.7 - 3.6 - 2.7 The list can be found here[1]. So, I propose introducing 2 new packages: - python3.7 - python3.6 And when we update the python package to 3.9: - python3.8 Does anyone have any big issue with this? What are your thoughts? [1] https://www.python.org/downloads/ Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Autumn extra cleaning
On Mon, 2020-10-05 at 07:16 +0200, Sven-Hendrik Haase via arch-dev- public wrote: > libusb-compat I can maintain this, please drop to [community] :) Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] [aur-general] AUR migration
On Mon, 2020-07-27 at 14:43 -1000, Gaetan Bisson via arch-dev-public wrote: > [2020-07-27 21:10:23 -0300] Giancarlo Razzolini: > > Em julho 27, 2020 21:03 Gaetan Bisson escreveu: > > > It's quite unsettling that we seem to be rushing to write a news post > > > while this very reasonable suggestion remains completely ignored. > > > > > > > It wasn't ignored. They keys were deliberately changed in the process. > > Why? Baptiste rightly points out "it's the same service as before and > (presumably) the host private keys were not compromised, so there is no > reason to change keys." Yet his message remains unanswered... If one machine gets compromised the keys are also compromised. If we can just use different keys on each machine to mitigate this, why wouldn't we? I think the short term bothers of changing the key do not warrant at all compromising security like this. But your experience might be different, is there anything in specific you are worried about or find annoying? I have been trying to figure what would possibly justify this but I can't, please let me know. Baptiste's answer was presumably under the assumption that the full machine would be migrated, but he would have to confirm. On which case, his request would be perfectly reasonable IMO. > > I think the issue you refer to happened on the orion -> gemini migration and > > You are correct. > > > I personally think that everything that runs as a service on Arch servers > > should > > be properly tracked on ansible, even if it's a user service. > > That is certainly a worthy goal but it does not imply that we must kill > everything that is not tracked by ansible at every migration. Copying > home directories over to the new host used to be standard practice for > any administrator of a system which serves multiple users... None of this happened, when it did hapen in soyuz everyone got properly notified and had plenty time to get their stuff out, on top of that, the system was backed up in case someone forgot. I don't understand what issue you are trying to get on here, Grazzolini already explained this did not happen. I agree with what you said, no machine should be killed without a proper handling of the user data, but what is the issue right now? Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] SDR package naming
On Fri, 2020-06-05 at 10:19 -0400, Eli Schwartz via arch-dev-public wrote: > On 6/5/20 9:04 AM, Filipe Laíns via arch-dev-public wrote: > > My main concern here is that it is not as simple as it just being > > Kyle's decision, it sets a precedent. I believe the naming is > > incorrect, and as such, should be fixed. I have tried initiating a > > conversation with the maintainer but with that didn't result in > > anything. > > It did result in something: he said "no". > > > I really don't want to step in anyone's toes, I have postponed this > > email as much as I could. Giving the lack of the reply from Kyle, one > > can only assume he does not care that much about the issue. I am fine > > with waiting one or two weeks before taking action to make sure he has > > time to reply, if there are no objections. > > "I don't agree with this, it fails to be memorable and using the > upstream shortname is confusing and does a disservice to users" sure > sounds like he objects to me. Hi Eli, Sorry, I wasn't clear, my bad. No consensus came from my attempt at contacting him. And there was no discussion, it was one sided, so I feel like this issue is not resolved. There are still relevant points that I want to see addressed. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] SDR package naming
On Thu, 2020-06-04 at 22:43 -1000, Gaetan Bisson via arch-dev-public wrote: > [2020-06-04 23:03:23 +0100] Filipe Laíns via arch-dev-public: > > 1) Rename libuhd to uhd > > 2) Use the gr- prefix instead of gnuradio- for GNURadio[2] blocks > > Your proposed changes indeed seem the correct thing to do, but Kyle > appears to have done a good job maintaining those packages over the > years. Do you really think it is worth bypassing his decision using a > community vote for something of almost no consequence? Hi Gaetan, My main concern here is that it is not as simple as it just being Kyle's decision, it sets a precedent. I believe the naming is incorrect, and as such, should be fixed. I have tried initiating a conversation with the maintainer but with that didn't result in anything. I really don't want to step in anyone's toes, I have postponed this email as much as I could. Giving the lack of the reply from Kyle, one can only assume he does not care that much about the issue. I am fine with waiting one or two weeks before taking action to make sure he has time to reply, if there are no objections. I have delayed bringing new GNURadio blocks to [community] due to the naming issue. I would like to have this resolved soon. > If we are to work together, we must sometimes learn to accept others' > decisions we don't 100% agree with... All I am trying to do is work together, but failing. As I said above, this decision affects more than Kyle's packages. Nobody is denying him of him opinion or decision rights, he is free to speak up at any time. I would be very happy to have a discussion with him about the issue :) Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] SDR package naming
Hi all,
I want to bring up two issues relating Kyle Keen's packages. I tried to
contact him and got an initial reply but I have not heard from him
since. It has been almost 4 months and I have sent him 3 different
followup emails. Since it was not super urgent I let a pretty
significant amount of time pass but I would like to have this resolved.
Here are the two issues I seek resolution:
1) Rename libuhd to uhd
UHD[1] is the upstream name. The software is pretty well known by that
name. I have only ever seen it being addressed as libuhd when talking
about the library it provides specifically, but even then UHD is the
standard. It also provides some tools alongside the library.
Following the Arch philosophy I think the package should be named
"uhd".
2) Use the gr- prefix instead of gnuradio- for GNURadio[2] blocks
The GNURadio ecosystem has a pretty well established prefix for
GNURadio blocks, which is gr-. Similarly to the previous issue, I have
never seen them being refereed as gnuradio-something. The upstream
names are gr-something. The gnuradio- prefix might seem to make things
more clear to new users, and even then I have my doubts, but it
definitely makes things more confusing for actual users of the project.
Previewing some counter-arguments to this, as I have discussed it with
people previously. This is totally different than, for eg., the python
ecosystem where we use the python- prefix. The gr- is given by the
upstream and is the *de facto* naming scheme for the project.
If you don't agree with the use of the gr- prefix, I think that
variation of the name should *at least* be present in provides.
These might seem like nitpicks, but I do very much care about them and
I believe they should be corrected.
You can find my attempt at communication with Kyle below.
I am very conflicted about this, but I think before I end the email I
should note that Kyle is packaging and he appears to have replied to
other team member(s). I am not sure if I have done something wrong or
behaved in an improper manner. I don't think I did, but if that's the
case please let me know.
[1] https://github.com/EttusResearch/uhd
[2] https://github.com/gnuradio
Cheers,
Filipe Laíns
On Thu, 2020-02-06 at 09:42 +, Filipe Laíns wrote:
> On Wed, 2020-02-05 at 23:21 -0500, keenerd wrote:
> > Sometimes my automatic email filing puts things in the wrong place,
> > but searching across all folders finds no emails from you about this.
> > How did you try to contact me before?
> >
> > UHD I am ambivalent on. However several other distros also call it
> > "libuhd" and it is a library after all.
> >
> > Gnuradio vs gr I don't agree with. A project needs to be very notable
> > for a two-letter naming convention. Python, for example, is
> > sufficient. Internally the Gnuradio project uses "gr" because, well,
> > they type it hundreds of times every day. Outside of their project,
> > it is less helpful and more confusing.
> >
> > There are several people deeply involved with Gnuradio who are also
> > Arch users. I more or less started packaging Gnuradio for them, and
> > their input was instrumental in making our packages the way they are.
> > They've never raised any of these issues.
> >
> > You did not provide any reasoning for why you think changes are necessary?
> >
> > -Kyle
> >
> > On 2/4/20, Filipe Laíns wrote:
> > > Hey Kyle,
> > >
> > > I think I have contacted out about this already but without much
> > > success. I would like to adjust the names of some SDR packages. I
> > > believe `libuhd` should be called `uhd` and packages that
> > > provide GNU
> > > Radio blocks should be named `gr-name` instead of `gnuradio-name`
> > > (this
> > > affects `gnuradio-{fcdproplus,iqbal,osmosdr}`).
> > >
> > > Can we adjust the naming? What do you think?
> > >
> > > Thank you,
> > > Filipe Laíns
> > >
>
> Sorry, my bad.
>
> Regarding uhd, that's what Debian does, not us. uhd is the upstream
> name and we should honor it. If you want examples just look at [1],
> from a quick browse I wasn't able to find any instances of packages
> where we add the lib prefix.
>
> Regarding GNU Radio blocks, it more of a grey area so I understand and
> respect your position. My reasoning is that the gr- prefix comes from
> the upstream, and AFAIK everybody else on earth uses it. When I started
> using Arch it was a bit confusing why the packages had different names,
> especially for a distribution that tries to keep as close to the
> upstream as possible. Do you think it would make sense to ask the GNU
> Radio guys what they think? Either way, it's ultimately your call.
> But if you don't want to rename it could we add the gr- prefixed name
> to provides/conflicts?
>
> Just some other unrelated things, I am bringing a few more SDR packages
> to Arch. I just pushed the necessary bits for Adalm-Pluto support, I am
> working on srsLTE (actually, almost finished, just waiting for some
> replies from the upstream) and I want
Re: [arch-dev-public] Packages spring cleanup!
On Sun, 2020-04-19 at 20:50 +0200, Jelle van der Waa wrote: > Hi all, > > I'm going to disown some packages as I no longer actively use them and I > want to shift focus into my on other Arch roles: > > ettercap > openttd-opensfx > openttd-opengfx > rdesktop > tesseract > tesseract-* > tidy > unshield > vim-pastie > vim-align > pstreams > pdf2djvu > bonnie++ > gsmartcontrol > leptonica > jbigkit > pbpst > ccd2iso > chromium-bsu > fsarchiver > dot2tex > > If packages need to be moved to [community] for you to be able to adopt > them, please say so. > > Greetings, > > Jelle I will take this opportunity to add some to the list: java-batik java-commons-io java-xmlgraphics-commons jxrlib Same reasoning, I no longer use them or have something that needs them. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Discussion - Increasing our CPU requirements
On Mon, 2020-03-30 at 09:07 +1000, Allan McRae via arch-dev-public wrote: > SSE4.2 is 2008 for Intel, 2011 for AMD. Though I guess some processors > were released without it for some time after that. AVX was released by > both in 2011. > > So why is one too far and the other not? I was looking at some edge cases where the CPUs had AVX but no SSE4.2. Intel's website is also a bit unreliable with older CPUs, a lot of the cases it is not listing extensions when it should be. > No need for pacman support. Just add higher instruction set to a new > repo and set that repo with higher priority. Right, that works too. Although, we can't have the word "auto" in the arch definition automatically identifying and enabling the supported extensions. > But that involves developers choosing which packages to build with > higher instruction sets, which requires extra developer time. Well. The thing about these CPU extensions is that they require very specific use workloads to be useful. They are more efficient when performing the tasks they were design to do, but they have an higher power consumption, which generally results in the CPU lowering clocks. To make proper use of them, you usually need to build your algorithm around them. Just enabling it in GCC might bring some performance improvements but they are usually *very* minimal. Upstreams that benefit from these extensions will most of the time write their own kernels and provide runtime detection or add a switch in the build system. I don't believe this is too much of burden for packagers. 99.9% of the time, just building a package with SEE/AVX natively in GCC will not bring any relevant performance enhancement. Furthermore, building an extra optimized version makes it much more reasonable for us to choose AVX2, which will have a much higher real world impact than just building everything with SSE4.2 instead. With that said, if you still want to bump the minimum requirements, could please consider taking just a little bit of time to run some benchmarks on packages you think should get an improvement? > Ideally, we would just autobuild for more optimized architectures, but > this requires auto-signing packages, which has not happened in the last > decade (but may in this one...). > > Picking an instruction set that is ~10 years old and making it the > default for the distro seems a reasonable approach to me. The instruction set might be 10 years old, but you have to look at the discontinuation date for CPU families which don't support it, and maybe add 1 or 2 years for the bulk of the inventory to be cleared out from stores. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Discussion - Increasing our CPU requirements
On Sun, 2020-03-29 at 16:25 +0100, Filipe Laíns via arch-dev-public wrote: > I would not be an entirely *It would > What I would for us to do is to create a x86-64-axv2, etc. that would *would like for us > let's me to believe (but this may be just me), I would be *let me to Ugh, sorry again. Today I am only catching the errors when obsessing over sending. signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Discussion - Increasing our CPU requirements
I want to clarify what I am proposing. I would not be an entirely new architecture in the sense of i686, CPU extensions are not different architectures and shouldn't be treated as such. What I would for us to do is to create a x86-64-axv2, etc. that would complement x86-64. We would not add it as a target for all packages, just for the ones that make sense. For this pacman would have to support architecture priority. We could have something like this: Architecture = x86-64-axv2 x86-64 This means if a x86-64-axv2 package is available, it would be selected over the x86-64 one. That way we don't need to rebuild all packages. My point here is that to me it does not really make sense to drop support for older CPUs. We will have little benefit in newer CPUs. Projects that need the performance already dynamically choose the CPU extensions to use in the runtime -- they will work on all x86-64 CPUs. If this did in fact bring a relevant performance improvement, like the original mail let's me to believe (but this may be just me), I would be all for it, but that's not the case. On Sun, 2020-03-29 at 16:51 +0200, Morten Linderud via arch-dev-public wrote: > Well, how do you think we supported two architectures? Why do you think > `extra-x86_64-build` is named the way it is? > > The "problem" is that we have no intentions of building 1 package 4 times and > keep things in sync by hand, it was tedious enough with i686, which was part > of > why it was dropped in the first place. Thus we want build-servers to do this > for > us. Then automate it? Is there any reason why we can't have the tooling build all architectures for us? Why not have an `extra-build` helper that will call extra-$arch-build for all every architecture? This will have practically the same affect as my SIMD packages not. The only difference would be how people are consuming them. It would just work out of the box instead of them having to install the -avx2 variant. > Allan is going to have a hard time argueing that the minimal improvements is > going to justify the absurd time we'll end up building things by hand, it's > the > crux of the problem essentially. I'm also sure he knows this. > > Surely we can bikeshed about which architectures to support, what we should > discuss is how we should accomplish the task in general. > > > Furthermore, if you do indeed whish to move this forward please present > > us with reasonable data. Take a few packages that would benefit from > > this, build them with the proposed architecture and show us benchmarks. > > I think it's gonna be very hard for you to find packages with > > considerable improvement but I might be wrong, please show me. > > See last paragraph. Which paragraph are you referring to? I will be taking a step back now. I will probably wait 1 or 2 days before replying. Feel free to reach me privately for a direct discussion if you want to. Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Discussion - Increasing our CPU requirements
On Sun, 2020-03-29 at 15:39 +0100, Filipe Laíns via arch-dev-public wrote: > I make the PKGBUILD build for 2 * I can make Sorry, I am a little distracted today. Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Discussion - Increasing our CPU requirements
On Sun, 2020-03-29 at 23:37 +1000, Allan McRae via arch-dev-public wrote: > On 29/3/20 11:17 pm, Filipe Laíns wrote: > > I would also like to note that rebuilding everything with forced > > support for AVX2 or whatever won't have much effect. Most packages do > > not have workloads where it would make use sense to use these CPU > > extensions, and as such, GCC would not use them. > > That assumes we just add AVX2. Whereas, requiring a CPU supporting AVX2 > would bring other optimizations that would be used. No, it should be true for all extensions. > As I replied earlier, AVX2 may be going too far. But is a good starting > point for discussion. If that is too far, what could we accept? > SSE4.2? AVX? Surely we can do better than pure x86_64. No, SSE4.2 is too far. For me, the minimum should be AVX. > To have a separate architecture would require automated builds, which > requires being able to sign packages automatically. And we have not > achieved database signing in 9 years I'm looking for a boost that > could be achieved now. No, it would not. Where is this coming from? I already build split packages with SIMD instructions, I make the PKGBUILD build for 2 architectures instead with a minimal patch. If pacman is not able to handle parallel architectures, we should fix that. I think it's a valid use case. Furthermore, if you do indeed whish to move this forward please present us with reasonable data. Take a few packages that would benefit from this, build them with the proposed architecture and show us benchmarks. I think it's gonna be very hard for you to find packages with considerable improvement but I might be wrong, please show me. Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Discussion - Increasing our CPU requirements
On Sun, 2020-03-29 at 20:26 +1000, Allan McRae via arch-dev-public wrote: > Remember when Arch Linux was optimized out of the box. We have the > blazingly fast i686 port while other distros hung out in i386 land. > Those were the days where the idea of Arch being fast started. Now it > has degraded to stuff of legend. > > Now, x86_64 is old. We should continue to push forward and add further > optimization. > > Reasonable optimizations to consider: > > AVX2 > FMA > SSE4.2 > > AVX2 is Intel Haswell and newer or AMD Ryzen and newer. This CPUs > released 2013 to 2015. So 5 - 7 years old. > > Discuss. Absolutely not! A huge amount of systems do not meet that requirements. This would rule out all Intel 3rd gen and older cpus, which would for eg. impact laptop models such as the Thinkpad X220 and Thinkpad X230. Instead of adding a requirement for newer CPU extensions let's fix this the proper way. The correct approach to this is to push upstreams to support dynamic detection of CPU extensions. That means the performance critical code is compiled with a different range of extensions and the binary detects at runtime what to use. A lot of them do that already, and for the ones which don't we should push for it (point them to [1]). Unfortunately, we will always have upstreams that don't support that. For this we should define a separate architecture. Right now I have building 2 variants of the projects (see srslte-avx2 and liquid-dsp- sse4.1), optimally we would have a different arch for them. I would also like to note that rebuilding everything with forced support for AVX2 or whatever won't have much effect. Most packages do not have workloads where it would make use sense to use these CPU extensions, and as such, GCC would not use them. There is only maybe a handful of packages in the repos that would benefit from this. [1] https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-target_005fclones-function-attribute Cheers, Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] [arch-events] FOSDEM Dinner 2020
On Mon, 2020-01-06 at 15:41 +0100, Morten Linderud via arch-events wrote: > Yo! > > Last year at FOSDEM we held a dinner with 15 people, some members from the > Arch > team and some users that wanted to join. It was a great event and people had a > nice dinner with some nice chats afterwards on various pubs. > > For this year I plan to do the same with some changes. First of all, we are > going to be holding it on saturday around 18:00 - 19:00 as opposed to friday. ^ Is this a typo? Isn't that too early? There will be talks until 19:00. In fact, I will be giving a talk from 18:30 to 18:55. > This is to ensure people are able to arrive friday evening, AND keep their > phones after their dinner. I'm still looking for mine :/ > members of the Arch team (developers, trusted users and support staff). Any > free > spots after this can be taken by anyone interested :). Send an email *offlist* > so I have some ability to keep track and inform you about further booking > details. > > The users requesting spots will get a headsup the week before FOSDEM to give > all > staff a chance to reply. Please ask me if you have any questions :) > > > --- > > I have again CC'ed arch-dev-public since I don't actually know how many people > watch arch-events (hinthint subscribe!). Sorry for the noise, and please do > tell > me if we are better off not cross-posting emails like these :) > > ___ > arch-events mailing list > arch-eve...@archlinux.org > https://lists.archlinux.org/listinfo/arch-events -- Filipe Laíns signature.asc Description: This is a digitally signed message part
Re: [arch-dev-public] Proposal: Build a ruleset for new packages and package quality
On Thu, 2019-12-12 at 13:21 +0100, Christian Rebischke via arch-dev-public wrote: > Hello everybody, > > Due to a longer discussion around alpha and beta packages in our > repositories in IRC yesterday, I would like to start a hopefully more > constructive discussion around this topic on the ML. > > Short summary for everybody what happened: > > I wanted to get caddy2.0.0-beta into [community], people started > discussing that we don't ship betas in our repositories and I should > either wait or ship the stable caddy v1 release. > > > I don't want to start this discussion around caddy again (I will wait > for a stable release). What I want to accomplish with this mail is: > > 1. find a consensus on rules which packages we allow in our repositories > and which don't. For the ones that were not in the discussion about caddy: The caddy devs consider 2.0.0-beta10 stable software but not stable API. From https://wiki.archlinux.org/index.php/Arch_package_guidelines#Package_versioning > Stable packages package stable releases I think this means stable software *and* stable API. Projects that depend on a certain library are not expected to be using beta releases. They will most likely only update once the library authors commit to a certain API. For most cases I don't think we should be packaging unstable APIs. > 2. find a consensus on rules on violating our rules regarding package > requirements. (e.g. What happens when TU/Dev XY violates our package > requirements? what is the punishment?) No punishments. As coderobe said, we should assume good faith. If someone is breaking the rule, ask them to fix it. If they continue to do it then I think it's means for disciplinary action, even removal if needed. I believe this should apply to both TUs and Devs. > 3. revive the discussion around better PKGBUILD quality (see also Eli's > thread about PKGBUILD quality on arch-tu: > https://lists.archlinux.org/private/arch-tu/2019-November/83.html) > > 4. I would like to have this rules written down somewhere, either in the > TU bylaws (in dev bylaws if this exists) or in a new format like > "community bylaws" (maybe we could combine this with the question about > a new leader). This should be in the packaging guidelines, not on some community or TU specific guilelines. I also think any sort of bylaws are not the correct place for this, there are to many exceptions for this to be correctly described in that kind of document. > 5. What do we do with the existing beta and alpha packages? Are they > granted asylum? Or do we remove them, to be consistent? > > extra libmspack 1:0.10.1alpha-2 > extra qt5-webkit 5.212.0alpha3-6 > community d-containers 0.8.0alpha.19-1 > extra frozen-bubble 2.2.1beta1-14 > extra hddtemp 0.3.beta15.53-1 > extra libcaca 0.99.beta19-2 > extra tsocks 1.8beta5-8 > community hydrogen 1.0.0beta1-1 > community lablgtk3 3.0.beta6-2 > community modclean 3.0.0beta.1-1 > community sdedit 4.2beta8-1 > community sniffit 0.3.7.beta-17 > community vbindiff 3.0_beta5-1 > community wqy-microhei 0.2.0_beta-9 > community wqy-microhei-lite 0.2.0_beta-9 Well, it depends on the case. As a general rule just don't package -beta. > 6. How do we define stable packages? beta or alpha is just a human made > word. I've seen devs who said their "1.0" version is unstable and > shouldn't be used in production. Should such a software get packaged? > And what about projects who use semantic versioning (the devs said so), > but they have a 0.* prefix release? True. We can't make rules based on version strings, each case needs to be analyzed. If the software is stable and the API is stable then package it. > 7. Another topic is a rule about "touching packages of others". In the > #archlinux-tu channel we came to the conclusion that TUs or devs should > ask the package maintainer before touching another devs/TUs package, how > do we want to handle this? Is this the consensus, or are touches without > prior question allowed? What do we do if somebody violates our rules? > etc This is the same as 2), assume good faith. If people abuse it then that should be grounds for disciplinary action. > 8. A few guys in the IRC pointed me to this guidelines in our wiki: > > Note: This page is only editable with Wiki-Admin/Dev Permission. > > https://wiki.archlinux.org/index.php/Arch_package_guidelines#Package_versioning > > My questions to this guidelines: > > 8.1. under which consensus where this rules defined? Are these rules the > result of a developer vote? of a leader decision? Or are they made up by > a few persons without consensus. > > 8.2 I can't find any list about punishments for violations of these > rules. > > 8.3 There is no documentation about our alpha and beta packages. I see > that there are exceptions for alpha and betas, but it's not clear for me > how these exceptions apply to the packages we have in our repositories. > > This needs to be documented, otherwise every person could push an
[arch-dev-public] I will be away this week
Hi, I will be away this week. I understand some of you have questions specifically about the new discord packages. Feel free to take any action that you feel necessary, delete, modify, etc..., just document that (email, irc or etc). Thabk you, Filipe Laíns 3DCE 51D6 0930 EBA4 7858 BA41 46F6 33CB B0EB 4BF2
Re: [arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions
On Sat, 2019-05-25 at 21:27 +1000, Allan McRae via arch-dev-public wrote:
> On 25/5/19 9:19 pm, Bruno Pagani via arch-dev-public wrote:
> > Hi,
> >
> > Le 25/05/2019 à 02:17, Filipe Laíns via arch-dev-public a écrit :
> > > I would also like to explore the idea of adding an "high performance"
> > > architecture which would be able to make use of SSE{,2,3,4,4.1,4.2} and
> > > AVX, which seem to be the standard for newer processors (>=2013). This
> > > would only be available for packages that do high performance computing
> > > (ex. openblas, sdrangel, etc.). Any thoughts on this?
> >
> > As said on IRC, they have been discussions before on having multiple
> > targets and corresponding repos, but the starting point is that we need
> > automated build before going into such a direction, and this in turn has
> > several requirements. I’ve linked to you the pad where we put our ideas
> > together regarding this.
> >
> > In the meantime, we had the case before of whether we should package
> > e.g. $pkgname-{sse4,avx} in a case where it mattered a lot, but it
> > turned out the software in question (embree) is able to do runtime
> > detection of available ISA. Maybe some other packages are doing this
> > too, else we could discuss whether allowing such flavours as a temporary
> > measure would be acceptable for selected packages.
>
> glibc detects available instruction sets and uses the best for many
> functions.
>
> I'd be very, very, very much against providing multiple variants of a
> package in our repos. Using asp and makepkg are is a hard solution for
> those who really need a few packages rebuilt.
>
> PS - I rebuilt [core] with -march=haswell recently as a test. Automated
> building is not an issue. Unattended package/database signing is the
> major stumbling block.
>
> Allan
In cases where the instruction set is detected at runtime it would not
be needed a new variation of the package since we can guarantee the
software isn't going to try to run any unsupported instructions.
What we are discussing really only applies to packages without runtime
SIMD code selection.
Thanks,
Filipe Laíns
3DCE 51D6 0930 EBA4 7858 BA41 46F6 33CB B0EB 4BF2
signature.asc
Description: This is a digitally signed message part
Re: [arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions
On Sat, 2019-05-25 at 13:19 +0200, Bruno Pagani via arch-dev-public wrote:
> Hi,
>
> Le 25/05/2019 à 02:17, Filipe Laíns via arch-dev-public a écrit :
> > I would also like to explore the idea of adding an "high performance"
> > architecture which would be able to make use of SSE{,2,3,4,4.1,4.2} and
> > AVX, which seem to be the standard for newer processors (>=2013). This
> > would only be available for packages that do high performance computing
> > (ex. openblas, sdrangel, etc.). Any thoughts on this?
>
> As said on IRC, they have been discussions before on having multiple
> targets and corresponding repos, but the starting point is that we need
> automated build before going into such a direction, and this in turn has
> several requirements. I’ve linked to you the pad where we put our ideas
> together regarding this.
>
> In the meantime, we had the case before of whether we should package
> e.g. $pkgname-{sse4,avx} in a case where it mattered a lot, but it
> turned out the software in question (embree) is able to do runtime
> detection of available ISA. Maybe some other packages are doing this
> too, else we could discuss whether allowing such flavours as a temporary
> measure would be acceptable for selected packages.
>
> Regards,
> Bruno
This is fine my me. My biggest concern was the fact C doesn't support
__attribute__(("instruction set here")) but there are of course
workarounds. Creating a new architecture only makes sense if there are
multiple packages needing this but it seems not. I am fine with a
suffix, although I was thinking more something like -simd as SSE4, AVX,
etc. are usually available at the same time. In this cases I think we
should add a post_install step that gives a warning if the user CPU
doesn't support the used instruction sets.
Thanks,
Filipe Laíns
3DCE 51D6 0930 EBA4 7858 BA41 46F6 33CB B0EB 4BF2
signature.asc
Description: This is a digitally signed message part
Re: [arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions
On Sat, 2019-05-25 at 10:35 +1000, Allan McRae wrote: > On 25/5/19 10:17 am, Filipe Laíns via arch-dev-public wrote: > > Hello, > > > > Currently there are no guidelines stating which x86 extensions (ex. > > SSE2, SEE3, SSE4, AVX, etc.) we support. This is a bit problematic > > since it lets compilers do what they want and possible generate code > > that can't run on some systems. > > > > Even though this is an issue, it's not complete anarchy, at least yet! > > Just kidding :p. The vast majority of our native packages are compiled > > with GCC and we do default to `-mtune=generic` which is good but not > > optimal. `-mtune=generic` tells GCC to compile for a generic processor > > so it's up to GCC to decide which architecture extensions would compose > > a generic processor. I haven't been able to find any documentation on > > what x86 extensions are enabled for a "generic" processor but I was > > able to track them down to MMX, SSE (or KNI) and SSE2. Being > > undocumented they could change at any time so I don't think we should > > rely on `-mtune=generic`. > > > > I think you need to look at the difference between -march and -mtune. > We use "-march=x86-64", which defines the instruction sets that can be > used. Adding "-mtune=generic" does not allow the inclusion of > additional instruction sets. > > Look at the output of: > gcc -march=x86-64 -Q --help=target > > Allan Yes! My bad. I got confused. From the `-march=x86_64` documentation: A generic CPU with 64-bit extensions. Setting `-mtune` to generic won't add any additional instruction sets by itself, but it does not prevent instruction sets from being added. Looks like GCC enables MMX, SSE and SSE2 by default, it isn't related at all to `-march` like I stated in the email but it still presents the same issue. What do you think? Filipe Laíns 3DCE 51D6 0930 EBA4 7858 BA41 46F6 33CB B0EB 4BF2 signature.asc Description: This is a digitally signed message part
[arch-dev-public] Create guidelines regarding SIMD instructions/x86 extensions
Hello,
Currently there are no guidelines stating which x86 extensions (ex.
SSE2, SEE3, SSE4, AVX, etc.) we support. This is a bit problematic
since it lets compilers do what they want and possible generate code
that can't run on some systems.
Even though this is an issue, it's not complete anarchy, at least yet!
Just kidding :p. The vast majority of our native packages are compiled
with GCC and we do default to `-mtune=generic` which is good but not
optimal. `-mtune=generic` tells GCC to compile for a generic processor
so it's up to GCC to decide which architecture extensions would compose
a generic processor. I haven't been able to find any documentation on
what x86 extensions are enabled for a "generic" processor but I was
able to track them down to MMX, SSE (or KNI) and SSE2. Being
undocumented they could change at any time so I don't think we should
rely on `-mtune=generic`.
What I propose is to define a set of x86 extensions to support and make
all compilers default to that if possible. I am fine with MMX, SSE and
SSE2 but they should be *our* choice, not GCC's.
This begs the question, would such approach be doable? So I ask the
compiler maintainers to please check if this would be possible.
I would also like to explore the idea of adding an "high performance"
architecture which would be able to make use of SSE{,2,3,4,4.1,4.2} and
AVX, which seem to be the standard for newer processors (>=2013). This
would only be available for packages that do high performance computing
(ex. openblas, sdrangel, etc.). Any thoughts on this?
Thanks,
Filipe Laíns
3DCE 51D6 0930 EBA4 7858 BA41 46F6 33CB B0EB 4BF2
signature.asc
Description: This is a digitally signed message part
[arch-dev-public] Mishandle of the 'Use gpg signatures and https sources' TODO list
Hello, In the process of updating the 'Use gpg signatures and https sources' TODO list, packages which weren't using https in their 'url' field were also added. This changes were reversed and a new TODO list was opened. Sorry for the trouble. Thanks, Filipe Laíns 3DCE 51D6 0930 EBA4 7858 BA41 46F6 33CB B0EB 4BF2 signature.asc Description: This is a digitally signed message part
