[Python-ideas] Re: "Curated" package repo?
On 2023-07-05 00:00, Christopher Barker wrote: I'm noting this, because I think it's part of the problem to be solved, but maybe not the mainone (to me anyway). I've been focused more on "these packages are worthwhile, by some definition of worthwhile). While I think Chris A is more focused on "which of these seemingly similar packages should I use?" -- not unrelated, but not the same question either. I noticed this in the discussion and I think it's an important difference in how people approach this question. Basically what some people want from a curated index is "this package is not junk" while others want "this package is actually good" or even "you should use this package for this purpose". I think that providing "not-junk level" curation is somewhat more tractable, because this form of curation is closer to a logical OR on different people's opinions. It may be that many people tried a package and didn't find it useful, but if at least one person did find it useful, then we can probably say it's not junk. Providing "actually-good level" curation or "recommendations" is harder, because it means you actually have to address differences of opinion among curators. Personally I tend to think a not-junk type curation is the better one to aim at, for a few reasons. First, it's easier. Second, it eliminates one of the main problems with trying to search for packages on pypi, namely the huge number of "mytestpackage1"-type packages. Third, this is what conda-forge does and it seems to be working pretty well there. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/XH2GTRKHZ2T4Z3VHQUCC5L7OATSHPUQU/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
On 2023-07-05 07:13, Chris Angelico wrote: Right; hence the question of how a "vetted Python package collection" would compare. I can type "sudo apt install python-" and add the name of a package, and I get some assurance that: 1) The package works 2) The package is useful enough 3) It's not malware 4) The specific*version* of the package works along with the versions of everything else. In my experience this is how conda-forge is too. The level of assurance is somewhat lower, but there is still a level of assurance about all those things. For point 4, the assurance is about the version you install working with the conda environment you install it into. This is an advantage over systemwide installs like debian packages because it means you can have multiple environments and know each one is consistent. Most of the problems arise when you circumvent conda's consistency checking, for instance by installing a package with pip rather than with conda. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/B6ASNFCW2K47UKXAWM3LWWH2UTKUPSUE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
Why not just use gpg signatures and maintain trusted signing keys? There’s no reason to reinvent the wheel. If a user wants to use a unsigned or untrusted packages, they have to accept the risk. Thanks, Greg On Wed, Jul 5, 2023 at 2:05 PM Chris Angelico wrote: > On Thu, 6 Jul 2023 at 03:57, James Addison via Python-ideas > wrote: > > I also agree with a later reply about avoiding the murkier side of > blockchains / etc. That said, it seems to me (again, sample size one > anecdata) that creating a more levelled playing field for package > publication could benefit from the use of some distributed technologies. > Even HTTP mirrors are, arguably, a basic form of that.. there's at least > one question related to recency of data, though. Delaying availability of > a package to an audience -- if it's important enough -- could under some > circumstances become effectively similar to censorship. > > > > A blockchain won't solve anything here. It would be completely and > utterly impractical to put the packages themselves into a blockchain, > so all you'd have is the index, and that means it's just a bad version > of PyPI's own single-page index. > > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/PTIS3HZHJSFV7ETWE7UP4HKXS4WN2OEO/ > Code of Conduct: http://python.org/psf/codeofconduct/ > ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/NYQSV7RO3GKE7272WZQ7VSIASNYKITMI/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
One way would be to categorise areas and sub-areas and have a clear indication, where the work has not been done. So that if I came to such place, I could find the sub-topic that I am interested in with clear indication of the status. > On 5 Jul 2023, at 21:48, Brendan Barnwell wrote: > > On 2023-07-04 17:21, Christopher Barker wrote: >> Anyway, I'd love to hear your thoughts on these ideas (or others?) - both >> technical and social. > > To my mind there are two interrelated social problems that make this > task difficult: > > 1) Promulgating a list of "good" packages tends to make people think packages > not on the list are not good (aka "implied exhaustiveness") > 2) In order to curate all or nearly all packages, you need curators with a > wide range of areas of interest and expertise (aka "breadth"). > > The reason these are interrelated is that once people start thinking > your list is exhaustive, it's really important to have breadth in the > curation, or else entire domains of utility can wind up having all packages > implicitly proscribed. > > As an example, a few months ago I wanted to do some automated email > manipulations via IMAP. I looked at the builtin imaplib module and found it > useless, so I went looking for other things. I eventually found one that > more or less met my needs (imap_tools). > > The question is, what happens when a person goes to our curated index > looking for an IMAP library? If they don't find one, does that mean there > aren't any, or there are but they're all junk, or just that there was no > curator who had any reason to explore the space of packages available in this > area? In short, it becomes difficult for a user to decide whether a tool's > *absence" from the index indicates a negative opinion or no opinion. > > There are ways around this, like adding categories (so if you see a > category you know someone at least attempted to evaluate some packages in > that category), but they can also have their own problems (like increasing > the level of work required for curation). I'm not sure what the best > solution is, but just wanted to mention this issue. > > -- > Brendan Barnwell > "Do not follow where the path may lead. Go, instead, where there is no path, > and leave a trail." > --author unknown > > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/WYLHRIBIRKP6W3HGCGFJGYHDL3GCSOR2/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/O233MP5I5MJLTTVK7FRIKJPD6736R3KX/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Double "at" operator for matmul exponentiation
> On 5 Jul 2023, at 21:07, Chris Angelico wrote: > > On Thu, 6 Jul 2023 at 04:02, Dom Grigonis wrote: >> What I would alternatively propose is to introduce a couple of (meaningless >> operators), so that library developers can make use of them as they wish. >> What characters aren't used? “$, ?, `” (or are they?). >> > > What should their precedences be? Operators have to be given that, at > least. Unfortunately, whatever precedence you choose, it's going to be > awkwardly wrong for at least some potential use-cases. Lowest precedence? a + b + c ?op? a / b / c a + b + (c ?op? a) / b / c As this in theory is the highest level operator, given it would mostly be used in framework level packages, it’s precedence naturally begs for the lowest status. But yes, then this would limit the operator to certain use cases. E.g. it wouldn’t be suitable to solve the problem of OP. > That said, though, there are some pretty cool packages out there that > create arbitrary operators. Some of them take advantage of multiple > operators to allow multiple effective precedences. Consider for > example: > https://pypi.org/project/infix/ Infix operator is where my though process began in the first place - see the e-mail. My proposal is exactly to have a free operator to be used in packages such as the one you pointed to. Ok, so Infix works in the way that operands can not recognise object in between operators, so that it falls to `dunders` of the object between operators. Although this works, there would be more flexibility if both the operands and operator class had `free` operator implementation space. Some new patterns might arise. It would be cleaner approach, currently `Infix` feels a bit hacky to me due to reusing of operators, which have another purpose. > ChrisA > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/YQDJ5UMH23FYL5FVFNBLWPOROD5ZZQOG/ > Code of Conduct: http://python.org/psf/codeofconduct/ ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/RVAKUXJ53QUBWFBGOSWTHP47HE3C4GCR/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
On 2023-07-04 17:21, Christopher Barker wrote: Anyway, I'd love to hear your thoughts on these ideas (or others?) - both technical and social. To my mind there are two interrelated social problems that make this task difficult: 1) Promulgating a list of "good" packages tends to make people think packages not on the list are not good (aka "implied exhaustiveness") 2) In order to curate all or nearly all packages, you need curators with a wide range of areas of interest and expertise (aka "breadth"). The reason these are interrelated is that once people start thinking your list is exhaustive, it's really important to have breadth in the curation, or else entire domains of utility can wind up having all packages implicitly proscribed. As an example, a few months ago I wanted to do some automated email manipulations via IMAP. I looked at the builtin imaplib module and found it useless, so I went looking for other things. I eventually found one that more or less met my needs (imap_tools). The question is, what happens when a person goes to our curated index looking for an IMAP library? If they don't find one, does that mean there aren't any, or there are but they're all junk, or just that there was no curator who had any reason to explore the space of packages available in this area? In short, it becomes difficult for a user to decide whether a tool's *absence" from the index indicates a negative opinion or no opinion. There are ways around this, like adding categories (so if you see a category you know someone at least attempted to evaluate some packages in that category), but they can also have their own problems (like increasing the level of work required for curation). I'm not sure what the best solution is, but just wanted to mention this issue. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/WYLHRIBIRKP6W3HGCGFJGYHDL3GCSOR2/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
On Thu, 6 Jul 2023 at 04:08, Gregory Disney wrote: > > Why not just use gpg signatures and maintain trusted signing keys? There’s no > reason to reinvent the wheel. If a user wants to use a unsigned or untrusted > packages, they have to accept the risk. > As an alternative to a blockchain? No idea, but I've never considered blockchains to be useful for anything more than toys anyway. As an alternative to a curated package list? That just comes down to who holds the trusted keys, so it's the same as the other suggestions, only you're looking at the mechanics for knowing whether it's on the list, as opposed to the mechanics for figuring out which things go on the list - two sides of the same coin, pretty much. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ANBP64KBYAB3MXO4NQDNMQHSXM525ZTN/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Double "at" operator for matmul exponentiation
On Thu, 6 Jul 2023 at 04:02, Dom Grigonis wrote: > What I would alternatively propose is to introduce a couple of (meaningless > operators), so that library developers can make use of them as they wish. > What characters aren't used? “$, ?, `” (or are they?). > What should their precedences be? Operators have to be given that, at least. Unfortunately, whatever precedence you choose, it's going to be awkwardly wrong for at least some potential use-cases. That said, though, there are some pretty cool packages out there that create arbitrary operators. Some of them take advantage of multiple operators to allow multiple effective precedences. Consider for example: https://pypi.org/project/infix/ ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/YQDJ5UMH23FYL5FVFNBLWPOROD5ZZQOG/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
On Thu, 6 Jul 2023 at 03:57, James Addison via Python-ideas wrote: > I also agree with a later reply about avoiding the murkier side of > blockchains / etc. That said, it seems to me (again, sample size one > anecdata) that creating a more levelled playing field for package publication > could benefit from the use of some distributed technologies. Even HTTP > mirrors are, arguably, a basic form of that.. there's at least one question > related to recency of data, though. Delaying availability of a package to an > audience -- if it's important enough -- could under some circumstances become > effectively similar to censorship. > A blockchain won't solve anything here. It would be completely and utterly impractical to put the packages themselves into a blockchain, so all you'd have is the index, and that means it's just a bad version of PyPI's own single-page index. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/PTIS3HZHJSFV7ETWE7UP4HKXS4WN2OEO/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Double "at" operator for matmul exponentiation
From perspective of calculation time of matrix multiplications Infix operators
is a reasonable solution to define a subset of your own.
https://doc.sagemath.org/html/en/reference/misc/sage/misc/decorators.html#sage.misc.decorators.infix_operator
The problem is that if one implements it, there has to be a substitution at the
expense of loosing a default operator. In scientific libraries this isn't great
as all existing operators tend to be used.
What I would alternatively propose is to introduce a couple of (meaningless
operators), so that library developers can make use of them as they wish. What
characters aren't used? “$, ?, `” (or are they?).
A $pow$ 3- meh
A ?pow? 3- looks ok. The question mark is appropriately marking
ambiguous/implementation dependent meaning.
A `pow` 3 - looks good
A @@ 3 - doesn’t look too great...
A.pow(3) - nice
If I was lacking just one operator (especially in such case where different
algorithms can be used), A.pow(-1, inv_type=‘LU') feels most convenient given
flexibility of arguments.
If I was to go heavy on operators in matrix algebra or defining some sort of
new syntax, then @@ wouldn’t help anyways - it’s just one (and fairly awful
looking).
> On 5 Jul 2023, at 13:02, ha...@interia.pl wrote:
>
>
> Python has the "star" ("*") operator for multiplication. In the context of
> collections it is supposed to mean element-wise multiplication. Its
> associated operator is __mul__. It also has the double star ("**") operator
> for exponentiation, which is repeated multiplication. Its associated operator
> is __exp__.
>
> In addition to that Python has the "at" ("@") operator for multiplication. In
> the context of collections it should mean linear product, like matrix times a
> matrix, or matrix times vector. Its associated method is __matmul__.
>
> For completeness we should now have the exponentiation operator meaning
> repeated application of the "at" operator. Its method could me __matexp__.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/AS3F4V4PCR4EUMYGC7HEECQHAFUVNU2M/
> Code of Conduct: http://python.org/psf/codeofconduct/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/F7ZNL3FCVY7HAXGB52D5HOOFG2RQFA7B/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
On Wed, Jul 5, 2023, 01:24 Christopher Barker wrote: > Stating a new thread with a correct title. > > On 2 Jul 2023, at 10:12, Paul Moore wrote: > > Unfortunately, too much of this discussion is framed as “someone should”, >> or “it would be good if”. No-one is saying “I will”. Naming groups, like >> “the PyPA should” doesn’t help either - groups don’t do things, people do. >> Who in the PyPA? Me? Nope, sorry, I don’t have the time or interest - I’d >> *use* a curated index, I sure as heck couldn’t *create* one. > > > Well, I started this topic, and I don't *think* I ever wrote "someone > should", and I certainly didn't write "PyPa should". > > But whatever I or anyone else wrote, my intention was to discuss what > might be done to address what I think is a real problem/limitation in the > discoverability of useful packages for Python. > > And I think of it not so much as "someone should" but as "it would be nice > to have". > > Of course, any of these ideas would take a lot of work to implement -- > and even though there are a lot of folks, on this list and elsewhere, > that would like to help, I don't think any substantial open-source project > has gotten anywhere without a concerted effort by a very small group (often > just 1) of people doing a lot of work to get it to a useful state before a > larger group can contribute. So I"m fully aware that nothings going to > happen unless *someone* really puts the work in up front. That someone > *might* be me, but I'm really good at over-committing myself, and not so > great at keeping my nose to the grindstone, so > > And I think this particular problem calls for a solution that would have > to be pretty well established before reaching critical mass to actually be > useful -- after all, we already have PyPi -- why go anywhere else that is > less comprehensive? > > All that being said, it's still worth having a conversation about what a > good solution might look like -- there are a lot of options, and hashing > out some of the ideas might inspire someone to rise to the occasion. > > The :problem", as I see it. > > - The Python standard library is not, and will never be fully > comprehensive -- most projects require *some* third party packages. > - There are a LOT of packages available on PyPi -- with a very wide range > of usefulness, quality and maintenance -- everything from widely used > packages with a huge community (e.g. numpy) to packages that are release > 0.0.1, and never seen an update, and may not even work. > > So the odds that there's a package that does what you need are good, but > it can be pretty hard to find them sometimes -- and can be a fair bit > of work to sift through to find the good ones -- and many folks don't feel > qualified to do so. > > This can result in two opposite consequences: > > 1) People using a package that really isn't reliable or maintained (or not > supported on all platforms, or ..) and getting stuck with it (I've had that > on some of my projects -- I'm pretty careful, but not everyone on my team > is) > > 2) People writing their own code - wasting time, and maybe not getting a > very good solution either. I've also had that problem on my projects... > > To avoid this -- SOME way for folks to find packages that have at least > has some level of vetting would be good -- exactly what level of vetting, > is a very open question, but I think "even a little" could be very helpful. > Doesn't each individual / team / company / organization already discuss and document their preferred packages, and (indirectly or directly) help to evolve them and consider alternatives by doing so? I think it's important that there's a common space where packages can exist and be available. Whether that realtime environment should state its' own preferred packages seems more debatable to me - because it could become a source of contention and gaming similar to search engine optimization. An exception could be software museums: I can see a curated collection of best-known and most-effective libraries used by particular cultures at points-in-time being of interest to future (and some current?) generations. I also agree with a later reply about avoiding the murkier side of blockchains / etc. That said, it seems to me (again, sample size one anecdata) that creating a more levelled playing field for package publication could benefit from the use of some distributed technologies. Even HTTP mirrors are, arguably, a basic form of that.. there's at least one question related to recency of data, though. Delaying availability of a package to an audience -- if it's important enough -- could under some circumstances become effectively similar to censorship. A few ideas that have come up in the previous thread -- more or less in > order of level of effort. > > 1) A "clearing house" of package reviews, for lack of a better word -- a > single web site that would point to various resources on the internet -- > blog posts, etc, that
[Python-ideas] Re: [Suspected Spam]"Curated" package repo?
Chris Angelico writes: > Part of the desired protection is the prevention of typosquatting. > That means there has to be something that you can point pip to and > say "install this package", and it's unable to install any > non-curated package. I think that the goalposts are walking though. How do you keep non-curated packages out of requirements.txt? Only if you have a closed ecosystem. Sounds like Anaconda or Condaforge or Debian to me, and people who want such a closed system should pick one-- and preferably only one --to support. The basic request as I understood it was to reduce what Chris Barker characterized as the cost of sifting through a maze of twisty little packages all alike, except that some are good, and some are bad, and some are downright ugly. Part of that is indeed to avoid typo- squatting malware. However, most of the squatters I'm aware of use names that look like improved or updated versions, and would not be frequently typoed. So my "click through to PyPI" approach would filter a majority, possibly a large majority, of non-curated packages. If people really want this somewhat draconian restriction to curated packages, fine by me (I'll stick to proofreading requirements.txt very carefully plus pip'ing from PyPI myself). I just don't see how it works or has advantages over existing options. Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/VYKKDCVPBESMKHVD2ORSDSPNULRVPIGW/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Double "at" operator for matmul exponentiation
None of the other repeated infix operators follow the same "repeated application" relation that * and ** have, i.e. // isn't repeated division, << and >> aren't repeated inequality comparisons (whathever that may be), and == isn't repeated assignment. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/X6JVECQNENCOWXVVAXMMOYXW2HXR365G/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Double "at" operator for matmul exponentiation
On Wed, 5 Jul 2023 at 11:18, wrote:
>
>
> Python has the "star" ("*") operator for multiplication. In the context of
> collections it is supposed to mean element-wise multiplication. Its
> associated operator is __mul__. It also has the double star ("**") operator
> for exponentiation, which is repeated multiplication. Its associated operator
> is __exp__.
>
> In addition to that Python has the "at" ("@") operator for multiplication. In
> the context of collections it should mean linear product, like matrix times a
> matrix, or matrix times vector. Its associated method is __matmul__.
>
> For completeness we should now have the exponentiation operator meaning
> repeated application of the "at" operator. Its method could me __matexp__.
This was discussed at the time of the PEP that introduced matmul:
https://peps.python.org/pep-0465/#non-definition-of-matrix-power
Quote:
"""
Earlier versions of this PEP also proposed a matrix power operator,
@@, analogous to **. But on further consideration, it was decided that
the utility of this was sufficiently unclear that it would be better
to leave it out for now, and only revisit the issue if – once we have
more experience with @ – it turns out that @@ is truly missed.
"""
It has been nearly 10 years which I guess is enough time to revisit
but most likely the numpy-discussion mailing list is the place to
start that discussion:
https://mail.python.org/mailman3/lists/numpy-discussion.python.org/
I think that the original authors wanted @@ and also other operators.
At the time there had been previous more ambitious PEPs proposing many
more operators or custom operators but those were rejected at least
partly for being too broad. As I remember it there was a conscious
effort to make PEP 465 as minimal as possible just to make sure that
at least the @ operator got accepted.
Matrix powers are not as widely used as matrix multiplication and
probably the most common use would be M@@-1 as a strange looking
version of M^-1 which would otherwise be spelled as np.linalg.inv(M).
It is usually a bad idea to compute matrix inverses though
(np.linalg.solve(a, b) is both faster and more accurate than
np.linalg.inv(a)*b). Matlab uses left and right division like a\b and
a/b for this but NumPy etc already define a/b to mean elementwise
division and then having a new operator a\b mean something completely
different from a/b might be confusing.
I think that the people who would have wanted to push forward with
things like this have probably reduced the scope of what seems
possible to the extent that the only thing left to do is to possibly
add @@. If that is all that is on the table then it possibly does not
have enough value to be worth the effort of a language change. On the
other hand it is a relatively small change since @ is already added.
--
Oscar
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/YKLPXMYCDOXBZ5AFC4WDDOF64RQPUT7M/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: [Suspected Spam]"Curated" package repo?
On Wed, 5 Jul 2023 at 17:12, Stephen J. Turnbull wrote: > > 4) A self contained repository of packages that you could point > > pip to -- it would contain only the packages that had met some > > sort of "vetting" criteria. In theory, anyone could run it, but > > a stamp of approval from the PSF would make it far more > > acceptable to people. This would be a LOT of work to get set > > up, and still a lot of work to maintain. > > Why "self-contained"? I always enter PyPI through the top page. I'd > just substitute curated-pypi.org's top page. Its search results would > be restricted to (or prioritize) the curated set, but it would take > me to the PyPI page of the recommended package. Part of the desired protection is the prevention of typosquatting. That means there has to be something that you can point pip to and say "install this package", and it's unable to install any non-curated package. There are many protections against typosquatting (and malware installation in general), but this particular one can be very effective, albeit with some fairly significant costs. ChrisA ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TUY6PLLPQOVDDKYV2CZE4QYAEKCZCURT/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
On Wed, 5 Jul 2023 at 17:00, Christopher Barker wrote: > I'm noting this, because I think it's part of the problem to be solved, but > maybe not the mainone (to me anyway). I've been focused more on "these > packages are worthwhile, by some definition of worthwhile). While I think > Chris A is more focused on "which of these seemingly similar packages should > I use?" -- not unrelated, but not the same question either. > Indeed, not the same question; but "some definition of worthwhile" is the crucial point here. If there is one single curated package index of "worthwhile" packages, who decides what's on it and what's not? If not everyone can agree, will there have to be multiple such listings? > Technically, conda is similar to pip -- it has a default "channel" (a channel > is an indexed repository of packages) it points to, and you can point it to a > different one, or any number of others, or install a single package from a > particular channel. > > Socially, it's pretty different > - There is no channel like PyPi that anyone can put anything on willy nilly. > - The default channel is operated by Anaconda.com -- and no one else can put > any thing on there. (they take suggestions, but it's a pretty big lift to get > them to add a package) > - The protocol for a channel is pretty simple -- all you really need is an > http server, but in practice, most folks host their channels on the > Anaconda.org server -- it's a free service that anyone can create a channel > on -- there are a LOT -- folks use them for their personal projects, etc. > So, high barrier to entry. Good to know. That's neither good nor bad inherently, but it is a point of note. > - Then there is conda-forge: > It grew out of an effort to collaborate among a number of folks operating > channels focused on particular fields -- met/ocean science, astronomy, > computational biology, ... we all had different needs, but they overlapped -- > why not share resources? Thanks to the heroic efforts of a few folks, it grew > to what it is now: a gitHub and CI -based conda package build system that > published a conda channel on anaconda.org with over 22,000 (wow! I think I'm > reading that right) packages. > > (https://anaconda.org/conda-forge/repo) > > They are curated -- anyone can propose a new package (via PR) -- but it only > gets added once it's been reviewed and approved by the core team. Curation > wasn't the goal, but it's necessary in order to have any hope that they will > all work together. The review process is really of the package, not the code > in the package (is it built correctly? is it compatible with the rest of > conda-forge? Does it include the license file? Is there a maintainer? ...) > But the end result is a fair bit of curation -- users can be assured that: > 1 - The package works > 2 - The package is useful enough that someone took the time to get it up > there. > 3 - It's very unlikely to be malware (I don't think the conda-forge policy > really looks hard for that, but typosquatting and that sort of thing are > pretty much impossible. > Cool. The trouble is, point 1 is nearly impossible to assure except in the very narrowest of definitions, and point 2's value correlates with the height of the barrier to entry, so it's a fairly strict tradeoff. And unless that barrier is extremely high, there will always be the possibility that someone puts in the effort to get malware pushed, although it does become vanishingly improbable. >> What about OS package managers like the Debian repositories? > > I have no idea, other than that the majors, at least, put a LOT of work into > having a pretty comprehensive base repository of "vetted" packages Right; hence the question of how a "vetted Python package collection" would compare. I can type "sudo apt install python-" and add the name of a package, and I get some assurance that: 1) The package works 2) The package is useful enough 3) It's not malware 4) The specific *version* of the package works along with the versions of everything else. This is a very strong set of criteria, much stronger than we'd be looking for here, as they come with correspondingly higher barriers to entry (getting a package update into the Debian repositories becomes harder and harder as the release date approaches). > conda-forge has about 22,121 -- that's enough to be very useful, but a lot of > use-cases are not well covered, and I know I still have to contribute one > once in a while. > > Looking now -- PyPi has 465,295 projects more than 20 times as many -- I > wonder how many of those are "useful"? Contrariwise, the Debian repository has under a thousand "python-*" packages, but with a much stronger requirement that they be useful. It's interesting that there are only twenty on PyPI for every one on conda-forge. I would have expected a greater ratio. It seems that conda-forge is able to be incomplete AND dauntingly large; how successful would you be at guessing
[Python-ideas] Re: "Curated" package repo?
I would go a bit further: DAOs are absolutely terrible for EVERYTHING, and anything that remotely mentions the acronym is a scam. Let's please, please, please not go down some cryptoscam, blockchain, rabbit hole here. Drop it, burn the remains, try to forget it ever happened. On Wed, Jul 5, 2023 at 3:57 AM Stephen J. Turnbull < turnbull.stephen...@u.tsukuba.ac.jp> wrote: > Christopher Barker writes: > > > Yes, it needs to be funded somehow, but some sort of donation / non > > profit / etc funding mechanism would be best -- but I don't think > > peer reviewers should be paid. Peer review in academic journals > > isn't cash compensated either. > > It's been done. The most common scheme is nominal compensation (say > USD50 per review) dependent on beating a relatively short deadline > (typically 1-3 months). But this is not really the same as academic > publishing. It's also not the same as movie and book reviewers who > are paid staffers (at least they used to be in the days of paper > journals). It has aspects of both. It might work here, although > funding and appointment of reviewers are tough issues. > > > I had to look that up: "Decentralized autonomous organization (DAO)" > > > > So, yes. > > Please, no. DAOs are fine when only money is at risk (too risky for > me, though). But they're a terrible way to manage a community or its > money. Too fragile, too inflexible. The history of DAOs is basically > an empirical confirmation of Arrow's Impossibility Theorem. > https://simple.wikipedia.org/wiki/Arrow%27s_impossibility_theorem > > ___ > Python-ideas mailing list -- python-ideas@python.org > To unsubscribe send an email to python-ideas-le...@python.org > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/ZLBEHQGMFIA5PR26XVDQF4YAVPIOYWY4/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- The dead increasingly dominate and strangle both the living and the not-yet born. Vampiric capital and undead corporate persons abuse the lives and control the thoughts of homo faber. Ideas, once born, become abortifacients against new conceptions. ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TAJCT2VGRWQG6WRVFL5EYOEYIOFK6ZJZ/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: Double "at" operator for matmul exponentiation
Every Python idea that has ever been proposed "for the sake of
completeness" has been rejected... at least in the 24 years I've been
following such closely.
Do you have an actual compelling use case? An abstract symmetry isn't going
to do it.
On Wed, Jul 5, 2023 at 6:17 AM wrote:
>
> Python has the "star" ("*") operator for multiplication. In the context of
> collections it is supposed to mean element-wise multiplication. Its
> associated operator is __mul__. It also has the double star ("**") operator
> for exponentiation, which is repeated multiplication. Its associated
> operator is __exp__.
>
> In addition to that Python has the "at" ("@") operator for multiplication.
> In the context of collections it should mean linear product, like matrix
> times a matrix, or matrix times vector. Its associated method is __matmul__.
>
> For completeness we should now have the exponentiation operator meaning
> repeated application of the "at" operator. Its method could me __matexp__.
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/AS3F4V4PCR4EUMYGC7HEECQHAFUVNU2M/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
The dead increasingly dominate and strangle both the living and the
not-yet born. Vampiric capital and undead corporate persons abuse
the lives and control the thoughts of homo faber. Ideas, once born,
become abortifacients against new conceptions.
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/U2XN5X67YZEAQOLCQZ4KRPPO5E3Z3QSE/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Double "at" operator for matmul exponentiation
Python has the "star" ("*") operator for multiplication. In the context of
collections it is supposed to mean element-wise multiplication. Its associated
operator is __mul__. It also has the double star ("**") operator for
exponentiation, which is repeated multiplication. Its associated operator is
__exp__.In addition to that Python has the "at" ("@") operator for
multiplication. In the context of collections it should mean linear product,
like matrix times a matrix, or matrix times vector. Its associated method is
__matmul__.For completeness we should now have the exponentiation operator
meaning repeated application of the "at" operator. Its method could me
__matexp__.___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at
https://mail.python.org/archives/list/python-ideas@python.org/message/AS3F4V4PCR4EUMYGC7HEECQHAFUVNU2M/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
Christopher Barker writes: > Yes, it needs to be funded somehow, but some sort of donation / non > profit / etc funding mechanism would be best -- but I don't think > peer reviewers should be paid. Peer review in academic journals > isn't cash compensated either. It's been done. The most common scheme is nominal compensation (say USD50 per review) dependent on beating a relatively short deadline (typically 1-3 months). But this is not really the same as academic publishing. It's also not the same as movie and book reviewers who are paid staffers (at least they used to be in the days of paper journals). It has aspects of both. It might work here, although funding and appointment of reviewers are tough issues. > I had to look that up: "Decentralized autonomous organization (DAO)" > > So, yes. Please, no. DAOs are fine when only money is at risk (too risky for me, though). But they're a terrible way to manage a community or its money. Too fragile, too inflexible. The history of DAOs is basically an empirical confirmation of Arrow's Impossibility Theorem. https://simple.wikipedia.org/wiki/Arrow%27s_impossibility_theorem ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZLBEHQGMFIA5PR26XVDQF4YAVPIOYWY4/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] [Suspected Spam]"Curated" package repo?
Christopher Barker writes: > So the odds that there's a package that does what you need are > good, but it can be pretty hard to find them sometimes -- and can > be a fair bit of work to sift through to find the good ones -- and > many folks don't feel qualified to do so. "Fair bit of work sifting" vs. "very hard work writing your own of higher quality" sounds like a VeryGoodDeal[tm] to me. As for "unqualified", if it's OK to be writing a program where you're unqualified to evaluate dependency packages, maybe it really doesn't matter if the package is best of breed? There are lots of programs where it doesn't matter, obviously. But if there's problem, it won't be solved by curation -- even best of breed is liable to be misused. > 4) A self contained repository of packages that you could point > pip to -- it would contain only the packages that had met some > sort of "vetting" criteria. In theory, anyone could run it, but > a stamp of approval from the PSF would make it far more > acceptable to people. This would be a LOT of work to get set > up, and still a lot of work to maintain. Why "self-contained"? I always enter PyPI through the top page. I'd just substitute curated-pypi.org's top page. Its search results would be restricted to (or prioritize) the curated set, but it would take me to the PyPI page of the recommended package. Why duplicate those pages? Would you also duplicate the storage? The only reason I can imagine for doing a "deep copy" would be to avoid the accusation of piggybacking on PyPI's and PyPA's hard work. Even if maintaining separate storage, many developers who use it would still depend on pip -- which gives priority to PyPI. They'd use the curated site to choose a package, but then just write its name in requirements.txt -- and of course that would work. So you don't even really avoid depending on PyPI for bandwidth (of course PyPI is going to spend on storage, anyway, so that doesn't count). > Personally, I think (4) is the best end result, but probably the > most work as well[*], so ??? Setup is pretty straightforward, maybe expensive if you go the self-contained route. But all the extra ongoing work is curation. And that's *really* hard to sustain. Riffing on "curated-pypi.org", I think it would be difficult to get "curated DOT pypi.org" for the reasons that have been repeatedly advanced, but why not your_team_name_here.curated.pypi.org? Set up a Patreon and get the TechDirt guy or somebody like that to write a monthly column reviewing the curators (once you get a lot, call it "featured curators", probably with the excuse that they specialize in some application field) or soliciting curators as the curated.pypi.org page. Steve ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/ZE4ZO4HZFJJCLAD3OOVMLRWI3WW6BLGD/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-ideas] Re: "Curated" package repo?
On Tue, Jul 4, 2023 at 5:49 PM Chris Angelico wrote: > On Wed, 5 Jul 2023 at 10:26, Christopher Barker > wrote: > > The :problem", as I see it. > > > > - The Python standard library is not, and will never be fully > comprehensive -- most projects require *some* third party packages. > > - There are a LOT of packages available on PyPi -- with a very wide > range of usefulness, quality and maintenance -- everything from widely used > packages with a huge community (e.g. numpy) to packages that are release > 0.0.1, and never seen an update, and may not even work. > > > > Remember though that this problem is also Python's strength here. The > standard library does not NEED to be comprehensive, and publishing on > PyPI is deliberately easy. The barrier to entry for the stdlib is > high; the barrier to entry for PyPI is low. > Absolutely -- and I"m not making any comment about the stdlib -- it's not the point here. But yes, the low (zero) barrier to entry to PyPi was probably the right way to go when it got started -- now, I'm not so sure. *some* barrier to entry would be helpful. > > https://wiki.python.org/moin/FrontPage Thanks, I had forgotten about the Wiki! It foundered for a while ( a lot of years ago -- I"ve been around...), but at a glance, it's looking good now. Imagine a page like > https://wiki.python.org/moin/Python_Package_Recommendations ... > That way, it's decentralized for editing, but has a central "hub" that > people can easily find. > yup -- great idea. > I suspect this would end up being broadly equivalent to the first > option, but with more effort by a core group of people (or a single > maintainer), and in return, would have a more consistent look and > feel. > yup. > > 3) A rating system built into PyPi -- This could be a combination of two > things: > > A - Automated analysis -- download stats, dependency stats, release > frequency, etc, etc, etc. > > B - Community ratings -- upvotes. stars, whatever. > > > > If done well, that could be very useful -- search on PyPi listed by > rating. However -- :done well" ios a huge challenge -- I don't think > there's a way to do the automated system right, and community scoring can > be abused pretty easily. But maybe folks smarter than me could make it work > with one or both of these approaches. > > > > Neither of them adequately answers questions like > "which is right *for this use-case*", I'm noting this, because I think it's part of the problem to be solved, but maybe not the mainone (to me anyway). I've been focused more on "these packages are worthwhile, by some definition of worthwhile). While I think Chris A is more focused on "which of these seemingly similar packages should I use?" -- not unrelated, but not the same question either. Which makes me realize that having a centralized package review site is complementary to a curated package index -- they are not replacements for one another. > 4) A self contained repository of packages that you could point pip to -- > ... Definitely possible; how would this compare to Conda? Technically, conda is similar to pip -- it has a default "channel" (a channel is an indexed repository of packages) it points to, and you can point it to a different one, or any number of others, or install a single package from a particular channel. Socially, it's pretty different - There is no channel like PyPi that anyone can put anything on willy nilly. - The default channel is operated by Anaconda.com -- and no one else can put any thing on there. (they take suggestions, but it's a pretty big lift to get them to add a package) - The protocol for a channel is pretty simple -- all you really need is an http server, but in practice, most folks host their channels on the Anaconda.org server -- it's a free service that anyone can create a channel on -- there are a LOT -- folks use them for their personal projects, etc. - Then there is conda-forge: It grew out of an effort to collaborate among a number of folks operating channels focused on particular fields -- met/ocean science, astronomy, computational biology, ... we all had different needs, but they overlapped -- why not share resources? Thanks to the heroic efforts of a few folks, it grew to what it is now: a gitHub and CI -based conda package build system that published a conda channel on anaconda.org with over 22,000 (wow! I think I'm reading that right) packages. (https://anaconda.org/conda-forge/repo) They are curated -- anyone can propose a new package (via PR) -- but it only gets added once it's been reviewed and approved by the core team. Curation wasn't the goal, but it's necessary in order to have any hope that they will all work together. The review process is really of the package, not the code in the package (is it built correctly? is it compatible with the rest of conda-forge? Does it include the license file? Is there a maintainer? ...) But the end result is a fair bit of curation -- users can be assured that: 1 -
[Python-ideas] Re: "Curated" package repo?
On Tue, Jul 4, 2023 at 10:06 PM Jonathan Crall wrote: > I like the idea of a vetted package index that pip can point to. The more > I think about it, the more I think that it needs some sort of peer review > system as the barrier to entry, and my thoughts to to establishing some > DeSci DAO that could distribute the peer review of packages amongst a set > of trusted maintainers while also being a mechanism to add new trusted > maintainers to the peer-review pool. > > Peer reviewers could be funded via a fee to submit a package for > publishing. > I agree until this point -- we REALLY don't want to have a pay to play system. Yes, it needs to be funded somehow, but some sort of donation / non profit / etc funding mechanism would be best -- but I don't think peer reviewers should be paid. Peer review in academic journals isn't cash compensated either. > I think to achieve a scalable, funded, decentralized, and trustworthy > package index a DAO makes some amount of sense. > I had to look that up: "Decentralized autonomous organization (DAO)" So, yes. -CHB -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/C2W5F6U5Y55SBGQZ5S3O3ZETSVYCOQYG/ Code of Conduct: http://python.org/psf/codeofconduct/
