[Python-Dev] Re: Merge Request Review Reminder
On 3/22/2021 8:51 PM, Senthil Kumaran wrote:
Given this discussion, I am motivated to review this and merge this
into stdlib. The conversation was helpful to me to understand for
utility value of the method introduced by the patch.
I had stated in the PR as well.
Thank you, both, Faisal and Jakub.
Yes, thank you.
I've also needed the "next network of a given prefix size" feature.
Eric
On Mon, Mar 22, 2021 at 5:40 PM Jakub Stasiak wrote:
On 22 Mar 2021, at 13:07, Faisal Mahmood wrote:
Thanks Jakub, I wasn't aware of the netaddr library but just gave it a play and
it does seem very similar and I think it's very useful and completely valid.
I think the subtle difference is also that my implementation allows you to
specify the next prefix as well, so it won't just find the next prefix of the
same size. This is a common use case when you are trying to find free address
spaces, you will need to look for networks of different sizes depending on what
you are doing.
Currently, you could say that you were given a network of 10.200.20.0/24 and
asked to split this network into a bunch of /26 networks, you can do this
easily using the subnets method:
list(IPv4Network("10.200.20.0/24").subnets(prefixlen_diff=2))
[IPv4Network('10.200.20.0/26'), IPv4Network('10.200.20.64/26'),
IPv4Network('10.200.20.128/26'), IPv4Network('10.200.20.192/26')]
That is very simple and effective, but not a very realistic example of how you
would split networks up. Given how limited the IPv4 address space is, normally
you may have to use that /24 block for multiple things, so can't just simply
split it up into /26's, you may need to instead get two /30's, one /27 and one
/25. Currently, I don't think there is any straightforward way to do this
without a 'next_network' method that I have implemented.
Example, given a network of 10.200.20.0/24, to get two /30's out of it, one /27
and one /25, I would do the following with my method:
first_network = IPv4Network("10.200.20.0/30")
# first_network = IPv4Network("10.200.20.0/30")
Then get the next one (note not specifying prefix just gives me another /30 -
i.e. same prefix size):
second_network = first_network.next_network()
# second_network = IPv4Network("10.200.20.4/30")
Then I would need to get the /27, so do this:
third_network = second_network.next_network(new_prefixlen=27)
# third_network = IPv4Network("10.200.20.32/27)
Finally the /25:
fourth_network = third_network.next_network(new_prefixlen=25)
# fourth_network = IPv4Network("10.200.20.128/25)
When you are dealing with the same prefix size for each new network, I think
it's just a simple case of adding 1 to the broadcast address each time, but
when you have different prefix sizes it's a bit more complicated.
On Sat, 20 Mar 2021 at 22:04, Jakub Stasiak wrote:
I don’t know if this is gonna support Faisal’s cause (“It’s used in a third
party library in the wild, that means stdlib could use it!”) or the exact
opposite (“it’s provided by a third party library, so may as well use third
party library and stdlib doesn’t necessarily need this”) but the quite popular
netaddr library that I’m a maintainer of does have IPNetwork.next() and
IPNetwork.previous() methods[1][2]. The main difference is that in netaddr:
* next() and previous() can produce networks arbitrary number of steps away,
not just the first closest network forwards or backwards
* going from a network to its supernetwork or subnetwork is done through a
separate set of supernet() and subnet() methods[3][4]
I can’t say how many library users actually consume this particular section of
the API, of course, but I expect this API to be used and it seems rather useful.
Best,
Jakub
[1] https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.next
[2] https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.previous
[3] https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.subnet
[4] https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.supernet
Thank you for the explanation – now I understand better how that prefix
parameter is useful.
It’s not *that* difficult to emulate this using netaddr, there’s an extra
switch-to-supernet or switch-to-subnet step but that’s about it:
from netaddr import IPNetwork
first_network = IPNetwork("10.200.20.0/30")
second_network = first_network.next()
second_network
IPNetwork('10.200.20.4/30')
third_network = second_network.supernet(27)[0].next()
third_network
IPNetwork('10.200.20.32/27')
fourth_network = third_network.supernet(25)[0].next()
fourth_network
IPNetwork('10.200.20.128/25’)
That said, if merging this into the Python stdlib doesn’t work out I’m open to
improving netaddr’s interface to better serve this use case.
Best wishes,
Jakub
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/l
[Python-Dev] Tests now start with only 131 imported modules, instead of 233
Hi, tl;dr Tests of the Python Test Suite now start with 131 imported modules, instead of 233. For example, asyncio, logging, multiprocessing and warnings are no longer imported by default. -- The Python test suite is run by a custom test runner called "libregrtest" (test.libregrtest). It can detect reference leaks, write the output as JUnit XML, run tests in parallel with multiple processes, etc. Tests are written with test.support which is a collection of helper functions. Over the years, test.support got more and more functions, and libregrtest got more and more features. The problem is that they import more and more modules. For example, "import test.support" imports 173 modules in Python 3.8! $ python3.8 >>> import sys, sys >>> before=set(sys.modules); import test.support; after=set(sys.modules) >>> len(after - before) 173 In these imports, you can find some "heavy" modules like asyncio and multiprocessing. Moreover, some modules have "side effects" on import. For example, "import logging" registers an "at fork" callback. In April 2020, I worked with Hai Shi on Python 3.10 in bpo-40275 to reduce the number of test.support imports from 171 to 25! test.support was splitted into multiple sub-modules: - bytecode_helper - hashlib_helper - import_helper - logging_helper - os_helper - script_helper - socket_helper - threading_helper - warnings_helper I continued the work in bpo-41718 to reduce the number of libregrtest imports, since libregrtest also imported many modules. A dummy test which does nothing in the master branch now only has 131 modules in sys.modules, whereas in Python 3.9 it has 233 (+102) modules! For example, asyncio, logging, multiprocessing and warnings are no longer imported by default. "import test.support" is now faster. master branch compared to Python 3.8 using ./python -c 'import test.support' command and Python built in release mode: [py38] 85.7 ms +- 6.8 ms -> [master] 32.4 ms +- 1.3 ms: 2.64x faster More realistic command running a single test method: ./python -m test test_os -m test_access [py38] 136 ms +- 5 ms -> [master] 97.8 ms +- 2.4 ms: 1.39x faster The only annoying point is that splitting test.support into sub-modules was not backported to Python 3.8 and 3.9, and so it will make backports a little bit more annoying. Sorry about that! -- If you know that two modules should be tested together, please write a dedicated test for that ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/I3OQTA3F66NQUN7CH2NHC5XZTO24QCIK/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Merge Request Review Reminder
Given this discussion, I am motivated to review this and merge this
into stdlib. The conversation was helpful to me to understand for
utility value of the method introduced by the patch.
I had stated in the PR as well.
Thank you, both, Faisal and Jakub.
On Mon, Mar 22, 2021 at 5:40 PM Jakub Stasiak wrote:
>
>
>
> > On 22 Mar 2021, at 13:07, Faisal Mahmood wrote:
> >
> > Thanks Jakub, I wasn't aware of the netaddr library but just gave it a play
> > and it does seem very similar and I think it's very useful and completely
> > valid.
> >
> > I think the subtle difference is also that my implementation allows you to
> > specify the next prefix as well, so it won't just find the next prefix of
> > the same size. This is a common use case when you are trying to find free
> > address spaces, you will need to look for networks of different sizes
> > depending on what you are doing.
> >
> > Currently, you could say that you were given a network of 10.200.20.0/24
> > and asked to split this network into a bunch of /26 networks, you can do
> > this easily using the subnets method:
> > >>> list(IPv4Network("10.200.20.0/24").subnets(prefixlen_diff=2))
> > [IPv4Network('10.200.20.0/26'), IPv4Network('10.200.20.64/26'),
> > IPv4Network('10.200.20.128/26'), IPv4Network('10.200.20.192/26')]
> >
> > That is very simple and effective, but not a very realistic example of how
> > you would split networks up. Given how limited the IPv4 address space is,
> > normally you may have to use that /24 block for multiple things, so can't
> > just simply split it up into /26's, you may need to instead get two /30's,
> > one /27 and one /25. Currently, I don't think there is any straightforward
> > way to do this without a 'next_network' method that I have implemented.
> >
> > Example, given a network of 10.200.20.0/24, to get two /30's out of it, one
> > /27 and one /25, I would do the following with my method:
> > >>> first_network = IPv4Network("10.200.20.0/30")
> > # first_network = IPv4Network("10.200.20.0/30")
> >
> > Then get the next one (note not specifying prefix just gives me another /30
> > - i.e. same prefix size):
> > >>> second_network = first_network.next_network()
> > # second_network = IPv4Network("10.200.20.4/30")
> >
> > Then I would need to get the /27, so do this:
> > >>> third_network = second_network.next_network(new_prefixlen=27)
> > # third_network = IPv4Network("10.200.20.32/27)
> >
> > Finally the /25:
> > >>> fourth_network = third_network.next_network(new_prefixlen=25)
> > # fourth_network = IPv4Network("10.200.20.128/25)
> >
> > When you are dealing with the same prefix size for each new network, I
> > think it's just a simple case of adding 1 to the broadcast address each
> > time, but when you have different prefix sizes it's a bit more complicated.
> >
> > On Sat, 20 Mar 2021 at 22:04, Jakub Stasiak wrote:
> >
> >
> >
> > I don’t know if this is gonna support Faisal’s cause (“It’s used in a third
> > party library in the wild, that means stdlib could use it!”) or the exact
> > opposite (“it’s provided by a third party library, so may as well use third
> > party library and stdlib doesn’t necessarily need this”) but the quite
> > popular netaddr library that I’m a maintainer of does have IPNetwork.next()
> > and IPNetwork.previous() methods[1][2]. The main difference is that in
> > netaddr:
> >
> > * next() and previous() can produce networks arbitrary number of steps
> > away, not just the first closest network forwards or backwards
> > * going from a network to its supernetwork or subnetwork is done through a
> > separate set of supernet() and subnet() methods[3][4]
> >
> > I can’t say how many library users actually consume this particular section
> > of the API, of course, but I expect this API to be used and it seems rather
> > useful.
> >
> > Best,
> > Jakub
> >
> > [1] https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.next
> > [2]
> > https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.previous
> > [3]
> > https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.subnet
> > [4]
> > https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.supernet
>
> Thank you for the explanation – now I understand better how that prefix
> parameter is useful.
>
> It’s not *that* difficult to emulate this using netaddr, there’s an extra
> switch-to-supernet or switch-to-subnet step but that’s about it:
>
> >>> from netaddr import IPNetwork
> >>> first_network = IPNetwork("10.200.20.0/30")
> >>> second_network = first_network.next()
> >>> second_network
> IPNetwork('10.200.20.4/30')
> >>> third_network = second_network.supernet(27)[0].next()
> >>> third_network
> IPNetwork('10.200.20.32/27')
> >>> fourth_network = third_network.supernet(25)[0].next()
> >>> fourth_network
> IPNetwork('10.200.20.128/25’)
>
> That said, if merging this into the Python stdlib doesn’t work out I’m open
> to improving netaddr’s interface to b
[Python-Dev] Re: Merge Request Review Reminder
> On 22 Mar 2021, at 13:07, Faisal Mahmood wrote:
>
> Thanks Jakub, I wasn't aware of the netaddr library but just gave it a play
> and it does seem very similar and I think it's very useful and completely
> valid.
>
> I think the subtle difference is also that my implementation allows you to
> specify the next prefix as well, so it won't just find the next prefix of the
> same size. This is a common use case when you are trying to find free
> address spaces, you will need to look for networks of different sizes
> depending on what you are doing.
>
> Currently, you could say that you were given a network of 10.200.20.0/24 and
> asked to split this network into a bunch of /26 networks, you can do this
> easily using the subnets method:
> >>> list(IPv4Network("10.200.20.0/24").subnets(prefixlen_diff=2))
> [IPv4Network('10.200.20.0/26'), IPv4Network('10.200.20.64/26'),
> IPv4Network('10.200.20.128/26'), IPv4Network('10.200.20.192/26')]
>
> That is very simple and effective, but not a very realistic example of how
> you would split networks up. Given how limited the IPv4 address space is,
> normally you may have to use that /24 block for multiple things, so can't
> just simply split it up into /26's, you may need to instead get two /30's,
> one /27 and one /25. Currently, I don't think there is any straightforward
> way to do this without a 'next_network' method that I have implemented.
>
> Example, given a network of 10.200.20.0/24, to get two /30's out of it, one
> /27 and one /25, I would do the following with my method:
> >>> first_network = IPv4Network("10.200.20.0/30")
> # first_network = IPv4Network("10.200.20.0/30")
>
> Then get the next one (note not specifying prefix just gives me another /30 -
> i.e. same prefix size):
> >>> second_network = first_network.next_network()
> # second_network = IPv4Network("10.200.20.4/30")
>
> Then I would need to get the /27, so do this:
> >>> third_network = second_network.next_network(new_prefixlen=27)
> # third_network = IPv4Network("10.200.20.32/27)
>
> Finally the /25:
> >>> fourth_network = third_network.next_network(new_prefixlen=25)
> # fourth_network = IPv4Network("10.200.20.128/25)
>
> When you are dealing with the same prefix size for each new network, I think
> it's just a simple case of adding 1 to the broadcast address each time, but
> when you have different prefix sizes it's a bit more complicated.
>
> On Sat, 20 Mar 2021 at 22:04, Jakub Stasiak wrote:
>
>
>
> I don’t know if this is gonna support Faisal’s cause (“It’s used in a third
> party library in the wild, that means stdlib could use it!”) or the exact
> opposite (“it’s provided by a third party library, so may as well use third
> party library and stdlib doesn’t necessarily need this”) but the quite
> popular netaddr library that I’m a maintainer of does have IPNetwork.next()
> and IPNetwork.previous() methods[1][2]. The main difference is that in
> netaddr:
>
> * next() and previous() can produce networks arbitrary number of steps away,
> not just the first closest network forwards or backwards
> * going from a network to its supernetwork or subnetwork is done through a
> separate set of supernet() and subnet() methods[3][4]
>
> I can’t say how many library users actually consume this particular section
> of the API, of course, but I expect this API to be used and it seems rather
> useful.
>
> Best,
> Jakub
>
> [1] https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.next
> [2]
> https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.previous
> [3] https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.subnet
> [4]
> https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.supernet
Thank you for the explanation – now I understand better how that prefix
parameter is useful.
It’s not *that* difficult to emulate this using netaddr, there’s an extra
switch-to-supernet or switch-to-subnet step but that’s about it:
>>> from netaddr import IPNetwork
>>> first_network = IPNetwork("10.200.20.0/30")
>>> second_network = first_network.next()
>>> second_network
IPNetwork('10.200.20.4/30')
>>> third_network = second_network.supernet(27)[0].next()
>>> third_network
IPNetwork('10.200.20.32/27')
>>> fourth_network = third_network.supernet(25)[0].next()
>>> fourth_network
IPNetwork('10.200.20.128/25’)
That said, if merging this into the Python stdlib doesn’t work out I’m open to
improving netaddr’s interface to better serve this use case.
Best wishes,
Jakub
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/NA422HQIN232FSCHSO3QQMGOPVZ4QM6W/
Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Python Language Summit 2021 Signups Are Now Open
Last call for signing up for Python Language Summit. The forms will automatically be closed at 6 AM Vancouver time tomorrow. Currently we have 82 sign ups and 22 topic proposals. Details: https://us.pycon.org/2021/summits/language/ Sign up stats: https://mariatta.ca/language_summit_data/ ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/SXO7GRPSCZDTQKJL5ROTAJZMFJ2EI3DL/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Merge Request Review Reminder
Thanks Jakub, I wasn't aware of the netaddr library but just gave it a play
and it does seem very similar and I think it's very useful and completely
valid.
I think the subtle difference is also that my implementation allows you to
specify the next prefix as well, so it won't just find the next prefix of
the same size. This is a common use case when you are trying to find free
address spaces, you will need to look for networks of different sizes
depending on what you are doing.
Currently, you could say that you were given a network of 10.200.20.0/24 and
asked to split this network into a bunch of /26 networks, you can do this
easily using the subnets method:
*>>> list(IPv4Network("10.200.20.0/24
").subnets(prefixlen_diff=2))*
*[IPv4Network('10.200.20.0/26' ),
IPv4Network('10.200.20.64/26' ),
IPv4Network('10.200.20.128/26' ),
IPv4Network('10.200.20.192/26' )]*
That is very simple and effective, but not a very realistic example of how
you would split networks up. Given how limited the IPv4 address space is,
normally you may have to use that /24 block for multiple things, so can't
just simply split it up into /26's, you may need to instead get two /30's,
one /27 and one /25. Currently, I don't think there is any straightforward
way to do this without a 'next_network' method that I have implemented.
Example, given a network of 10.200.20.0/24, to get two /30's out of it, one
/27 and one /25, I would do the following with my method:
*>>> first_network = IPv4Network("10.200.20.0/30 ")*
*# first_network = IPv4Network("10.200.20.0/30 ")*
Then get the next one (note not specifying prefix just gives me another /30
- i.e. same prefix size):
*>>> second_network = first_network.next_network()*
*# second_network = IPv4Network("10.200.20.4/30 ")*
Then I would need to get the /27, so do this:
*>>> third_network = second_network.next_network(new_prefixlen=27)*
*# third_network = IPv4Network("10.200.20.32/27 )*
Finally the /25:
*>>> fourth_network = third_network.next_network(new_prefixlen=25)*
*# fourth_network = IPv4Network("10.200.20.128/25
)*
When you are dealing with the same prefix size for each new network, I
think it's just a simple case of adding 1 to the broadcast address each
time, but when you have different prefix sizes it's a bit more complicated.
On Sat, 20 Mar 2021 at 22:04, Jakub Stasiak wrote:
>
>
> > On 20 Mar 2021, at 04:39, Guido van Rossum wrote:
> >
> > Honestly this is an example of a module that would have been better off
> outside the stdlib.
> >
> > On Fri, Mar 19, 2021 at 14:43 Senthil Kumaran
> wrote:
> > On Mon, Feb 15, 2021 at 8:36 AM Faisal Mahmood
> > wrote:
> > >
> > > Hello,
> > >
> > > I hope you are all well, I currently have an issue / merge request
> that has become stale and as per the guidelines I am sending this e-mail to
> request someone to review it for me please.
> > >
> > > Issue Number: 42861 (https://bugs.python.org/issue42861)
> > > Pull Request: 24180 (https://github.com/python/cpython/pull/24180)
> > >
> >
> > Could you share some motivation references that could help explain why
> > this next_network method will be helpful? Is this common enough to
> > warrant a method in the stdlib IP-Address module?
> > If there is prior-art in other external libraries or libraries of
> > other languages, that will be helpful to know and review too.
> >
> > I had looked at this a month ago when you pinged for review, but I
> > could not immediately determine the benefit of having this method in
> > stdlib, irrespective of implementation correctness or API Signature.
> > I also lack extensive experience with ipv6.
> >
> > Thanks,
> > Senthil
>
>
> I don’t know if this is gonna support Faisal’s cause (“It’s used in a
> third party library in the wild, that means stdlib could use it!”) or the
> exact opposite (“it’s provided by a third party library, so may as well use
> third party library and stdlib doesn’t necessarily need this”) but the
> quite popular netaddr library that I’m a maintainer of does have
> IPNetwork.next() and IPNetwork.previous() methods[1][2]. The main
> difference is that in netaddr:
>
> * next() and previous() can produce networks arbitrary number of steps
> away, not just the first closest network forwards or backwards
> * going from a network to its supernetwork or subnetwork is done through a
> separate set of supernet() and subnet() methods[3][4]
>
> I can’t say how many library users actually consume this particular
> section of the API, of course, but I expect this API to be used and it
> seems rather useful.
>
> Best,
> Jakub
>
> [1]
> https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.next
> [2]
> https://netaddr.readthedocs.io/en/latest/api.html#netaddr.IPNetwork.previous
> [3]
> https://
[Python-Dev] Fwd: Re: aiter/anext review request
On Sat, Mar 20, 2021 at 8:03 PM Guido van Rossum wrote: > The operator module is also C. > (In my 2018 PR, I was able to add the Python implementations of `aiter` and `anext` to Lib/operator.py, right above where it does "from _operator import *" toward the bottom of the file. I interpreted this pattern there to mean "shadow Python implementations with C implementations where available, but otherwise it's okay to use the Python implementations above". Is that right?) Dan also suggested freezing the Python implementations into the binary as a way of keeping their implementations in Python, but in builtins rather than operator. > I am pleading to remove the 2nd arg to aiter, which should simplify the > code. > Done in the latest revision of https://github.com/python/cpython/pull/23847. That does indeed simplify the patch significantly, nice.[1] All checks are passing. Yury or any other core developers, would you like to merge this now? Thanks again, everyone. --Josh, Dan, and Justin [1] The latest revision adds the following to the `aiter` docs: Unlike the :func:`iter` builtin, :func:`aiter` has no 2-argument variant. Often, this variant can be replaced with assignment expressions:: while chunk := await sock.read(CHUNK_SIZE): ... I think when I originally worked on this in 2018, PEP 572 had only just been approved. :-) ___ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/JN3QVRDD3GZILCL7HTC6PPQL5N7J2NSA/ Code of Conduct: http://python.org/psf/codeofconduct/
