[Python-Dev] [RELEASE] Python 3.7.4rc1 and 3.6.9rc1 are now available

[Ned Deily]

Python 3.7.4rc1 and 3.6.9rc1 are now available. 3.7.4rc1 is the release
preview of the next maintenance release of Python 3.7, the latest feature
release of Python. 3.6.9rc1 is the release preview of the first
security-fix release of Python 3.6. Assuming no critical problems are
found prior to 2019-06-28, no code changes are planned between these
release candidates and the final releases. These release candidates are
intended to give you the opportunity to test the new security and bug
fixes in 3.7.4 and security fixes in 3.6.9. We strongly encourage you to
test your projects and report issues found to bugs.python.org as soon as
possible. Please keep in mind that these are preview releases and, thus,
their use is not recommended for production environments.

You can find the release files, a link to their changelogs, and more
information here:
https://www.python.org/downloads/release/python-374rc1/
https://www.python.org/downloads/release/python-369rc1/

--
  Ned Deily
  n...@python.org -- []
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7R7C7YLPXJ6AR3M2XZA7IKSHZ2SQFBYG/

[Python-Dev] Re: radix tree arena map for obmalloc

[Tim Peters]

[Tim]
> - For truly effective RAM releasing, we would almost certainly need to
> make major changes, to release RAM at an OS page level.   256K arenas
> were already too fat a granularity.

We can approximate that closely right now by using 4K pools _and_ 4K
arenas:  one pool per arena, and mmap()/munmap() are then called on
one page at a time.

[Don't try this at home ;-)  There are subtle assumptions in the code
that there are at least two pools in an arena, and those have to be
overcome first.]

For memcrunch.py, using 200x the original initial objects, this works
quite well!  Note that this still uses our current release-arenas
heuristic:  the only substantive change from the status quo is setting
ARENA_SIZE to POOL_SIZE (both 4 KiB - one page).

# arenas allocated total   =  873,034
# arenas reclaimed =  344,380
# arenas highwater mark=  867,540
# arenas allocated current =  528,654
528654 arenas * 4096 bytes/arena   =2,165,366,784

# bytes in allocated blocks=1,968,234,336
# bytes in available blocks=  141,719,280
5349 unused pools * 4096 bytes =   21,909,504
# bytes lost to pool headers   =   25,118,640
# bytes lost to quantization   =8,385,024
# bytes lost to arena alignment=0
Total  =2,165,366,784

So, at the end, space utilization is over 90%:

1,968,234,336 / 2,165,366,784 = 0.90896117

OTOH, an even nastier version of the other program I posted isn't
helped much at all, ending like so after phase 10:

# arenas allocated total   =1,025,106
# arenas reclaimed =   30,539
# arenas highwater mark=1,025,098
# arenas allocated current =  994,567
994567 arenas * 4096 bytes/arena   =4,073,746,432

# bytes in allocated blocks=  232,861,440
# bytes in available blocks=2,064,665,008
424741 unused pools * 4096 bytes   =1,739,739,136
# bytes lost to pool headers   =   27,351,648
# bytes lost to quantization   =9,129,200
# bytes lost to arena alignment=0
Total  =4,073,746,432

So space utilization is under 6%:

232,861,440 / 4,073,746,432 = 0.0571615

Believe it or not, that's slightly (but _only_ slightly) better than
when using the current 256K/4K arena/pool mix, which released no
arenas at all and ended with

232,861,440 / 4,199,022,592 = 0.05545611

utilization.

So:

- There's substantial room for improvement in releasing RAM by
tracking it at OS page level.

- But the current code design is (very!) poorly suited for that.

- In some non-contrived cases it wouldn't really help anyway.

A natural question is how much arena size affects final space
utilization for memcrunch.py.  Every successive increase over one pool
hurts, but eventually it stops mattering much.  Here are the possible
power-of-2 arena sizes, using 4K pools, ending with the smallest for
which no arenas get reclaimed:

1,968,234,336 / 2,165,366,784 = 0.90896117
528654 arenas * 4096 bytes/arena   =2,165,366,784
# bytes in allocated blocks=1,968,234,336

1,968,234,336 / 2,265,399,296 = 0.86882447
276538 arenas * 8192 bytes/arena   =2,265,399,296
# bytes in allocated blocks=1,968,234,336

1,968,235,360 / 2,441,314,304 = 0.80621957
149006 arenas * 16384 bytes/arena  =2,441,314,304
# bytes in allocated blocks=1,968,235,360

1,968,235,360 / 2,623,799,296 = 0.75014707
80072 arenas * 32768 bytes/arena   =2,623,799,296
# bytes in allocated blocks=1,968,235,360

1,968,235,360 / 2,924,216,320 = 0.67308131
44620 arenas * 65536 bytes/arena   =2,924,216,320
# bytes in allocated blocks=1,968,235,360

1,968,235,360 / 3,299,475,456 = 0.59652978
25173 arenas * 131072 bytes/arena  =3,299,475,456
# bytes in allocated blocks=1,968,235,360

1,968,235,360 / 3,505,913,856 = 0.56140437
13374 arenas * 262144 bytes/arena  =3,505,913,856
# bytes in allocated blocks=1,968,235,360

1,968,235,360 / 3,552,051,200 = 0.55411233
6775 arenas * 524288 bytes/arena   =3,552,051,200
# bytes in allocated blocks=1,968,235,360

1,968,235,360 / 3,553,624,064 = 0.55386707
3389 arenas * 1048576 bytes/arena  =3,553,624,064
# bytes in allocated blocks=1,968,235,360

Most of the damage was done by the time we reached 128K arenas, and
"almost all" when reaching 256K.

I expect that's why I'm not seeing much of any effect (on arena
recycling effectiveness) moving from the current 256K/4K to the PR's
1M/16K.  256K/4K already required "friendly" allocation/deallocation
patterns for the status quo to do real good, and 256K 

[Python-Dev] Re: Who uses libpython38.a on Windows?

[Shahrokh Mortazavi]

oops, apologies - pls ignore .


Sent from Outlook


From: Shahrokh Mortazavi
Sent: Tuesday, June 18, 2019 2:16 PM
To: Steve Dower; Python Dev
Subject: Re: [Python-Dev] Who uses libpython38.a on Windows?

ok.

currently it has develop for web, ML, scripting, edu.
how about another box: "Building apps for Windows using Python"...  is that a 
reality yet?




Sent from Outlook


From: Steve Dower 
Sent: Friday, June 14, 2019 1:53 PM
To: Python Dev
Subject: [Python-Dev] Who uses libpython38.a on Windows?

One of the most annoying steps in building the Windows installers is
generating the libpython38.a file. It's annoying, because it requires
having "generic enough" MinGW tools to ensure that the file is
compatible with whatever version of MinGW might be trying to build
against the regular Windows distribution.

I would like to stop shipping this file in 3.8 and instead put the steps
into the docs to show people how to generate them themselves (with the
correct version of their tools):

gendef python38.dll > tmp.def
dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a
-m i386:x86-64

(Obviously the commands themselves are not complicated if you already
have gendef and dlltool, but currently a normal CPython build system
does not have these.)

Before just doing this, I wanted to put out a request for information:

* Do you rely (or know anyone who relies) on libpython38.a on Windows?
* Are you able to add the two commands above to your build? If not, why not?

Thanks,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman3%2Flists%2Fpython-dev.python.org%2Fdata=02%7C01%7Csmortaz%40exchange.microsoft.com%7Ca5552da04ce84bba15d208d6f10ba4d2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961429645522465sdata=4U1L3ErhvsdvqNFHw1tRA1WSflHeDCmYtnveW94jmvs%3Dreserved=0
Message archived at 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Farchives%2Flist%2Fpython-dev%40python.org%2Fmessage%2FBYU35PWDNJ54COLNCFCSY3MCFYPF4KUK%2Fdata=02%7C01%7Csmortaz%40exchange.microsoft.com%7Ca5552da04ce84bba15d208d6f10ba4d2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961429645522465sdata=joltfcNqYned18OyebJOzkdJRG%2Ff%2BHPGlzzxzyE19jQ%3Dreserved=0
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VIFPK4KVIY4TZCX5LR3EV5P6KP6OEK37/

[Python-Dev] Re: Who uses libpython38.a on Windows?

[Shahrokh Mortazavi]

ok.

currently it has develop for web, ML, scripting, edu.
how about another box: "Building apps for Windows using Python"...  is that a 
reality yet?




Sent from Outlook


From: Steve Dower 
Sent: Friday, June 14, 2019 1:53 PM
To: Python Dev
Subject: [Python-Dev] Who uses libpython38.a on Windows?

One of the most annoying steps in building the Windows installers is
generating the libpython38.a file. It's annoying, because it requires
having "generic enough" MinGW tools to ensure that the file is
compatible with whatever version of MinGW might be trying to build
against the regular Windows distribution.

I would like to stop shipping this file in 3.8 and instead put the steps
into the docs to show people how to generate them themselves (with the
correct version of their tools):

gendef python38.dll > tmp.def
dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a
-m i386:x86-64

(Obviously the commands themselves are not complicated if you already
have gendef and dlltool, but currently a normal CPython build system
does not have these.)

Before just doing this, I wanted to put out a request for information:

* Do you rely (or know anyone who relies) on libpython38.a on Windows?
* Are you able to add the two commands above to your build? If not, why not?

Thanks,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman3%2Flists%2Fpython-dev.python.org%2Fdata=02%7C01%7Csmortaz%40exchange.microsoft.com%7Ca5552da04ce84bba15d208d6f10ba4d2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961429645522465sdata=4U1L3ErhvsdvqNFHw1tRA1WSflHeDCmYtnveW94jmvs%3Dreserved=0
Message archived at 
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Farchives%2Flist%2Fpython-dev%40python.org%2Fmessage%2FBYU35PWDNJ54COLNCFCSY3MCFYPF4KUK%2Fdata=02%7C01%7Csmortaz%40exchange.microsoft.com%7Ca5552da04ce84bba15d208d6f10ba4d2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636961429645522465sdata=joltfcNqYned18OyebJOzkdJRG%2Ff%2BHPGlzzxzyE19jQ%3Dreserved=0
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CRV4HAFP5KMYBDIU4LHZXKZVGVPEI5UT/

[Python-Dev] Re: why is not 64-bit installer the default download link for Windows?

[Steve Dower]

On 18Jun2019 1025, Stephen J. Turnbull wrote:

Oleg Broytman writes:
  > On Tue, Jun 18, 2019 at 10:09:59AM -, smartmanoj42...@gmail.com wrote:

  > > Why don't we check the architecture using js and provide the
  > > appropriate version?
  >
  >Because the downloading computer is not necessary the installation
  > target.

Sure, but (a) it's a good bet, and (b) somebody downloading to install
on a different machine is more likely to know what they're doing and
be conscious of issues of platform.


Equally, someone more conscious of the issues will know to go and get 
the 64-bit version if they explicitly want it. But for practically 
everyone the 32-bit version will be just fine.


There's no definitive answer to this, which means regardless of which 
decision we make we will have to continue to explain it over and over 
again. Right now, status quo and the lack of a volunteer to update the 
web site means that sticking with the 32-bit link is easier to explain 
than having to figure out why a particular machine was offered a 
particular download when it is not correct.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BTXH437DIA7PSWSF7JVMJBCEL37JFSG2/

[Python-Dev] Re: radix tree arena map for obmalloc

[Tim Peters]

And one more random clue.

The memcrunch.py attached to the earlier-mentioned bug report does
benefit a lot from changing to a "use the arena with the smallest
address" heuristic, leaving 86.6% of allocated bytes in use by objects
at the end (this is with the arena-thrashing fix, and the current
256K/4K arena/pool sizes).  Which isn't surprising, else the person
who opened that report wouldn't have attached that specific script ;-)

However, when I increase its number of starting objects by a factor of
200, changing the heuristic barely helped at all.  It managed to
reclaim a few dozen more arenas by the end (out of 13,556 total arenas
allocated), but left only 56.1% of arena space in use by objects.

For the later program I posted, which by accident-rather-than-design
is even worse for arena cycling, changing the heuristic didn't help at
all.  Either way, no arenas were released (although one empty arena is
retained in the usable_arenas list), and less than 10% of arena space
was in use at the end of phase 10.

A few things to note, then:

- For truly effective RAM releasing, we would almost certainly need to
make major changes, to release RAM at an OS page level.   256K arenas
were already too fat a granularity.

- Whether a program benefits from what we do now appears to rely by
far most on its patterns of allocation/deallocation, rather than on
the arena-releasing heuristic OR on the arena size.

- While the by-lowest-address heuristic is obviously more effective in
one known artificial program so far ;-) , it's more expensive than
what we do now.  That's a full-blown sorting problem.  Ordering by
number of free pools is not:  that's essentially a bucket-distribution
problem, since the number of _possible_ values is small.  The data
structure changes I already checked in truly take constant time, in
all cases, to restore sorted order when an arena's number of free
pools changes.  When sorting by address, the best that can be done in
general is to take O(log(A)) time to insert a new arena (where A is
the number of arenas).  And a linked list is no good for that either
(unless, e.g., it's made fancier, like a skip list).

The quick experiment I tried used one-at-time search to put an arena
in the by-address-ordered list (the submitted patch also did), and we
already know from experience that can become a timing disaster when
hundreds of thousands of arenas are in use.  OTOH, that only needs to
be done once each time an arena is added to usable_arenas, not every
time an arena's number of free pools changes.

So, overall:

- I'm not inclined to pursue changing the arena-freeing heuristic.

- Changes to keep track of OS page-level use would likely require
starting over from scratch.  But would also make using far larger
pools practical (on 64-bit boxes).

- I still haven't found a program where changing from 256K to 1M
arenas makes a lick of appreciable difference in arena recycling
effectiveness:  for a given program chewing on a given problem size,
"does pretty well" or "does horribly" has been the outcome either way.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/F2LEVIFQL55EVSCCIBKDFSPPTG3KY3P2/

[Python-Dev] Re: radix tree arena map for obmalloc

[Antoine Pitrou]

On Mon, 17 Jun 2019 13:44:29 -0500
Tim Peters  wrote:
> 
> To illustrate, I reverted that change in my PR and ran exactly same
> thing.  Wow - _then_ the 1M-arena-16K-pool PR reclaimed 1135(!) arenas
> instead of none.  Almost all worse than uselessly.  The only one that
> "paid" was the last:  the run ended with 3361 arenas still in use
> instead of 3362.  Because with the change, one entirely empty arena
> remained on the usable_arenas list.
> 
> So, word to the wise:  when looking at _debugmallocstats() output, like:
> 
> # arenas allocated total   =4,496
> # arenas reclaimed =1,135
> # arenas highwater mark=3,362
> # arenas allocated current =3,361
> 3361 arenas * 1048576 bytes/arena  =3,524,263,936
> 
> the number "reclaimed" isn't really telling us much:  before 3.9, it
> may be telling us only how many times obmalloc wasted time on useless
> arena thrashing.

That's a very nice improvement :-)

Regards

Antoine.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BS4XUDVOVTB7CECA6UHWWXCUFECGAWAV/

[Python-Dev] Re: why is not 64-bit installer the default download link for Windows?

[Stephen J. Turnbull]

Oleg Broytman writes:
 > On Tue, Jun 18, 2019 at 10:09:59AM -, smartmanoj42...@gmail.com wrote:

 > > Why don't we check the architecture using js and provide the
 > > appropriate version?
 > 
 >Because the downloading computer is not necessary the installation
 > target.

Sure, but (a) it's a good bet, and (b) somebody downloading to install
on a different machine is more likely to know what they're doing and
be conscious of issues of platform.

It's a common practice nowadays, probably for that reason.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GQY7NA7Q4UEBQPDA4UDU37M2I4KDLYOW/

[Python-Dev] Re: python3 -bb and hash collisions

[Christian Heimes]

On 18/06/2019 18.32, Daniel Holth wrote:
> set([u"foo", b"foo]) will error because the two kinds of string have the
> same hash, and this causes a comparison. Is that correct?

Yes, it will fail with -bb, because it turns comparison between str and
bytes into an error. This can also happen with other strings when
hash(u'somestring') & mask == hash(b'otherbytes') & mask. The mask of a
set starts with PySet_MINSIZE - 1 == 8 and increases over team.

Christian


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZIF2MRBWSMSCFP6E7PZOBI5KYP46QZPK/

[Python-Dev] Re: why is not 64-bit installer the default download link for Windows?

[Oleg Broytman]

On Tue, Jun 18, 2019 at 10:09:59AM -, smartmanoj42...@gmail.com wrote:
> Why don't we check the architecture using js and provide the appropriate 
> version?

   Because the downloading computer is not necessary the installation
target.

Oleg.
-- 
Oleg Broytmanhttps://phdru.name/p...@phdru.name
   Programmers don't die, they just GOSUB without RETURN.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GHFN5GVXPU7RPA7MLOPFNWXBPSKBIGUF/

[Python-Dev] python3 -bb and hash collisions

[Daniel Holth]

set([u"foo", b"foo]) will error because the two kinds of string have the
same hash, and this causes a comparison. Is that correct?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MRKTBTY7ZTU3I6SCWRHB533O2QMYA4VR/

[Python-Dev] Re: why is not 64-bit installer the default download link for Windows?

[Steve Dower]

On 18Jun2019 0309, smartmanoj42...@gmail.com wrote:

why is not 64-bit installer the default download link for Windows?


Because there are still 32-bit operating systems out there, and the 
32-bit version of Python will work just fine on both 32 and 64-bit 
Windows (and will use less memory than the 64-bit version).


The version we publish to the Microsoft Store is only available in 
64-bit, as it has a much higher minimum OS version requirement and the 
proportion of 32-bit installs drops off dramatically. (That download 
link was removed from the website while the PSF figures out whether they 
are allowed to link to it.)



Why don't we check the architecture using js and provide the appropriate 
version?


I'm sure the team working on https://github.com/python/pythondotorg 
would love to have more contributors :)


From my point of view, I want the link to be the same on any machine, 
so I'm quite happy with it being static.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5XT3YU3XACBYYAQMRXSKILNO5H3ED3GG/

[Python-Dev] Re: why is not 64-bit installer the default download link for Windows?

[smartmanoj42857]

Why don't we check the architecture using js and provide the appropriate 
version?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/NBVAUQOD42SGKSGVJLG3GRYSL6YVZ24V/