[issue29515] socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows

[Jan Gosmann]

Change by Jan Gosmann <jan+pythontrac...@hyper-world.de>:


--
nosy: +jgosmann

___
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue29515>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: cPickle fails on manually compiled and executed Python function

[Jan Gosmann]

On 07/18/2017 01:07 AM, dieter wrote:

"Jan Gosmann" <j...@hyper-world.de> writes:


[...]
fn = load_pyfile('fn.py')['fn']
[...]

"pickle" (and "cpickle") are serializing functions as so called
"global"s, i.e. as a module reference together with a name.
This means, they cannot handle functions computed in a module
(as in your case).
Note that I'm assigning the computed function to a global/module level 
variable. As far as I understand the documentation 
<https://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled> 
that should be all that matters because only the function name will be 
serialized.

I am quite convinced that "pickle" will not be able to deserialize (i.e. load)
your function (even though it appears to perform the serialization
(i.e. dump).
Actually the deserialization works fine with either module. That is both 
pickle.loads(pickle.dumps(fn)) and cPickle.loads(pickle.dumps(fn)) give 
me back the function.


By now I realized that a pretty simple workaround works. Instead of 
doing `fn = load_pyfile('fn.py')['fn']` the following function 
definition works with both pickle modules:


_fn = load_pyfile('fn.py')['fn']
def fn(*args, **kwargs):
return _fn(*args, **kwargs)

--
https://mail.python.org/mailman/listinfo/python-list

cPickle fails on manually compiled and executed Python function

[Jan Gosmann]

Hi,

today I came across some weird behaviour (a bug?) in Python 2.7.13 (on 
Linux) with the cPickle module. The pickle module works and so does the 
pickle module in Python 3.


I have a file fn.py with a minimal function definition:

```
def fn():
pass
```

The actual code that I run is in a separate file (test.py):

```
import cPickle
import pickle

def load_pyfile(filename):
source = ''
with open(filename, 'r') as f:
source += f.read()
code = compile(source, filename, 'exec')
loaded = {'__file__': filename}
exec(code, loaded)
return loaded

fn = load_pyfile('fn.py')['fn']

print(pickle.dumps(fn))
print('')
print(cPickle.dumps(fn))
```

The first print works fine, but the one with cPickle leads to an 
exception. Here is the output:


```
c__main__
fn
p0
.

Traceback (most recent call last):
  File "test.py", line 17, in 
print(cPickle.dumps(fn))
TypeError: expected string or Unicode object, NoneType found
```

I don't understand why the cPickle module behaves differently in this 
case. Is this expected? And if so, how do I fix it? Or can this be 
considered a bug? (In that case I could open an issue in the bug 
tracker.)


Cheers, Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 03/30/2017 02:19 PM, INADA Naoki wrote:

FYI, this small patch may fix your issue:
https://gist.github.com/methane/8faf12621cdb2166019bbcee65987e99

I can verify that the patch fixes the issue for me.

Do you still need more information about the `transitive_closure` 
function and my usage of sets and frozensets? Or any other way I can be 
helpful in fixing this? (There are a few questions in this thread that I 
haven't answered so far, but as the problem seems to be identified it 
might not be worth spending time on that.)


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 03/29/2017 11:31 PM, Chris Angelico wrote:

On Thu, Mar 30, 2017 at 2:19 PM, Rick Johnson
 wrote:

[...]
Really? How could your clients not notice 60 GB of memory
usage unless they are running some kind of mad-dog insane
top-of-the-line hardware? (Got Benjamins???) Of course, in
the case they are not DARPA scientist supported by a
viturally unlimited supply of tax dollars provided by the
endless toils of American slave-bots, how could they ignore
the thrashing? o_O

Did you read the project's README? This is a dramatic reduction from
the normal memory usage of this kind of job. So, yes, they *are* going
to have top-of-the-line hardware. If you're doing a job that normally
would require 256GB of RAM, and instead of being able to run in 40GB,
it needs 60GB, are you going to notice? You probably have 64GB or
128GB.

ChrisA
Actually, it is not memory, but time improvements. Most people won't run 
models of the size of Spaun, so usually the memory requirements will be 
much lower


Jan
--
https://mail.python.org/mailman/listinfo/python-list

[issue29949] sizeof set after set_merge() is doubled from 3.5

[Jan Gosmann]

Changes by Jan Gosmann <jan+pythontrac...@hyper-world.de>:


--
nosy: +jgosmann

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue29949>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

That's great news. I'm busy with other things right now, but will look 
into your findings in more detail later.



On 03/30/2017 02:09 PM, INADA Naoki wrote:

Filed an issue: https://bugs.python.org/issue29949

Thanks for your report, Jan.

On Fri, Mar 31, 2017 at 3:04 AM, INADA Naoki  wrote:

Maybe, this commit make this regression.

https://github.com/python/cpython/commit/4897300276d870f99459c82b937f0ac22450f0b6

Old:
minused = (so->used + other->used)*2  (L619)

New:
minused = so->used + other->used  (L620)
minused = (minused > 5) ? minused * 2 : minused * 4;  (L293)

So size of small set is doubled.

$ /usr/bin/python3
Python 3.5.2+ (default, Sep 22 2016, 12:18:14)
[GCC 6.2.0 20160927] on linux
Type "help", "copyright", "credits" or "license" for more information.

import sys
s = set(range(10))
sys.getsizeof(frozenset(s))

736
$ python3
Python 3.6.0 (default, Dec 30 2016, 20:49:54)
[GCC 6.2.0 20161005] on linux
Type "help", "copyright", "credits" or "license" for more information.

import  sys
s = set(range(10))
sys.getsizeof(frozenset(s))

1248


On Fri, Mar 31, 2017 at 2:34 AM, INADA Naoki  wrote:

I reproduced the issue.
This is very usual, memory usage issue.  Slashing is just a result of
large memory usage.

After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
30GB on Python 3.6.
And Python 3.6 starts slashing in 2nd optimization pass.

I enabled tracemalloc while 1st pass.  Results is end of this mail.
It seems frozenset() cause regression, but I'm not sure yet.
I don't know what is contents of frozenset yet.  (I know almost
nothing about this application).

Jan, do you know about what this is?
Could you make script which just runs `transitive_closure(edges)` with
edges similar to
`log_reduction.py spaun`?

I'll dig into it later, maybe next week.

---
Python 3.6.1
1191896 memory blocks: 22086104.2 KiB
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
 reachables[vertex] = frozenset(reachables[vertex])
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
 self.dependents = transitive_closure(self.dg.forward)
602986 memory blocks: 51819.1 KiB
   File "", line 14
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
 first_view=None, v_offset=0, v_size=0, v_base=None)

Python 3.5.3
1166804 memory blocks: 6407.0 KiB
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
 reachables[vertex] = frozenset(reachables[vertex])
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
 self.dependents = transitive_closure(self.dg.forward)
602989 memory blocks: 51819.3 KiB
   File "", line 14
   File 
"/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
 first_view=None, v_offset=0, v_size=0, v_base=None)


--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 29 Mar 2017, at 20:12, Steve D'Aprano wrote:

If you can demonstrate this effect using simple example code without 
the
external dependencies (using nothing but the standard library) and 
people

can replicate it, I think it should be reported as a bug.


I probably won't be able to demonstrate the effect with simple example 
code because I haven't the slightest idea what is causing it. Also, I'm 
not sure how much more time I want to invest in this. After all it is a 
problem that might never get noticed by any users of the software.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 28 Mar 2017, at 14:21, INADA Naoki wrote:

On Wed, Mar 29, 2017 at 12:29 AM, Jan Gosmann <j...@hyper-world.de> 
wrote:


I suppose smaller and faster benchmark is better to others looking for 
it.

I already stopped the azure instance.
[...]
There are no maxrss difference in "smaller existing examples"?
[...]
I want to investigate RAM usage, without any swapping.


Running further trials indicate that the problem actually is related to 
swapping. If I reduce the model size in the benchmark slightly so that 
everything fits into the main memory, the problem disappears. Only when 
the memory usage exceeds the 32GB that I have, Python 3.6 will acquire 
way more memory (from the swap) than Python 3.5.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 28 Mar 2017, at 6:11, INADA Naoki wrote:


I managed to install pyopencl and run the script.  It takes more than
2 hours, and uses only 7GB RAM.
Maybe, some faster backend for OpenCL is required?

I used Microsoft Azure Compute, Standard_A4m_v2 (4 cores, 32 GB
memory) instance.


I suppose that the computing power of the Azure instance might not be 
sufficient and it takes much longer to get to the phase where the memory 
requirements increase? Have you access to the output that was produced?


By the way, this has nothing to do with OpenCL. OpenCL isn't used by the 
log_reduction.py script at all. It is listed in the dependencies because 
some other things use it.



More easy way to reproduce is needed...


Yes, I agree, but it's not super easy (all the smaller existing examples 
don't exhibit the problem so far), but I'll see what I can do.



My best idea about what's going on at the moment is that memory
fragmentation is worse in Python 3.6 for some reason. The virtual 
memory
size indicates that a large address space is acquired, but the 
resident
memory size is smaller indicating that not all of that address space 
is
actually used. In fact, the code might be especially bad to 
fragmentation
because it takes a lot of small NumPy arrays and concatenates them 
into
larger arrays. But I'm still surprised that this is only a problem 
with

Python 3.6 (if this hypothesis is correct).

Jan


Generally speaking, VMM vs RSS doesn't mean fragmentation.
If RSS : total allocated memory ratio is bigger than 1.5, it may be
fragmentation.
And large VMM won't cause swap.  Only RSS is meaningful.


I suppose you are right that from the VMM and RSS numbers one cannot 
deduce fragmentation. But I think RSS in this case might not be 
meaningful either. My understanding from [the Wikipedia description] is 
that it doesn't account for parts of the memory that have been written 
to the swap. Or in other words RSS will never exceed the size of the 
physical RAM. VSS is also only partially useful because it just gives 
the size of the address space of which not all might be used?


Anyways, I'm getting a swap usage of about 30GB with Python 3.6 and 
zsh's time reports 2339977 page faults from disk vs. 107 for Python 3.5.


I have some code to measure the unique set size (USS) and will see what 
numbers I get with that.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 28 Mar 2017, at 3:08, Peter Otten wrote:

> Perhaps numpy's default integer type has changed (assuming you are using
> integer arrays, I did look at, but not into your code)?
>
> You could compare
>
 numpy.array([42]).itemsize
> 8
>
> for the two interpreters.

Both report 8 for integer and float point arrays.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 20:12, Chris Angelico wrote:

On Tue, Mar 28, 2017 at 11:00 AM, Chris Angelico  
wrote:

In any case, I've installed nvidia-opencl-dev and it seems to be
happy. How long should I expect this to run for?

Now testing under CPython 3.7.



It ran for about 14 minutes, then memory usage spiked and went into
the page file. Used roughly 8GB of RAM prior to that point, I think?
Unfortunately, I don't think my hard disk is fast enough for this to
run through the page file in any reasonable time.


Yeah, it will be problematic with less than 32GB.

My latest findings: The zsh `time` built-in reports a maximum of only 
30GB used which seems to correspond to the resident memory size, while 
the virtual memory size exceeds 40GB for the process and the total 
memory allocated over all processes exceeds 60GB (with the main 
contribution from the process that supposedly only uses ~40GB).


My best idea about what's going on at the moment is that memory 
fragmentation is worse in Python 3.6 for some reason. The virtual memory 
size indicates that a large address space is acquired, but the resident 
memory size is smaller indicating that not all of that address space is 
actually used. In fact, the code might be especially bad to 
fragmentation because it takes a lot of small NumPy arrays and 
concatenates them into larger arrays. But I'm still surprised that this 
is only a problem with Python 3.6 (if this hypothesis is correct).


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 20:00, Chris Angelico wrote:


By the way, since you're aiming this at recent Python versions, you
could skip the 'virtualenv' external dependency and use the built-in
'venv' package:

$ python3 -m venv env


Yeah, I know about venv. The last time I tried it, there was still some 
issue that prevented me from using it and I'm also still targeting 2.7 
too.



I'm a little confused here. When I install PIP requirements, it wants
to set up OpenCL, but your README seems to be contrasting your
implementation with the OpenCL one. Or is that for the sake of
benchmarking, so you can compare one against the other?


Right, the PIP requirements are more than actually required for this 
particular problem. But it is indeed for benchmarking comparing 
different backends for a neural network simulator.


For the log_reduction.py script the nengo==2.3.1 and numpy==1.12.0 are 
probably sufficient.



In any case, I've installed nvidia-opencl-dev and it seems to be
happy. How long should I expect this to run for?


On an Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz with 32GB RAM it 
probably takes about 30 minutes with Python 3.5; longer with 3.6 because 
things get written to the swap partition.


Jan
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 18:42, Chris Angelico wrote:


Are you able to share the program? I could try it on my system and see
if the same thing happens.


Yes, it is on GitHub (use the fixes branch): 
https://github.com/ctn-archive/gosmann-frontiers2017/tree/fixes

Installation instructions are in the readme.
The command I'm running is `python scripts/log_reduction.py spaun`

I was looking at the output of htop to see the memory consumption. The 
`time` built-in of the zsh might report a lower memory usage (I'm 
running it right now to make sure), which would also be surprising. But 
I'm certain that Python 3.6 uses more memory because the run time 
increases due to using the swap partition.


If you run the program, it will take an initial 10–15 minutes of 
processing during which the memory usage should increase to roughly 
10GB. Then the output “[INFO] Optimizing model...” will be printed 
after which the memory usage continues to increase to around 30GB (maybe 
a bit more with Python 3.6). This constitutes a first “optimization 
pass” that will be ended with the message “[INFO] Pass 1 [views]: 
Reduced 996917 to 913666 operators in 62.519006s.” Then in the second 
optimization pass the memory consumption will continue to increase with 
Python 3.6 (up to ~60GB being consumed, where less than 5GB should be 
due to other processes, though the virtual and resident memory reported 
for the Python process are only ~42GB and ~30GB respectively) and in 
later passes it will start decreasing again. With Python 3.5 the memory 
consumption stays about 30GB and decreases in later passes.


Let me know if you have more questions.
--
https://mail.python.org/mailman/listinfo/python-list

Re: Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

On 27 Mar 2017, at 18:30, Peter Otten wrote:


Are you perchance comparing 32-bit Python 3.5 with 64-bit Python 3.6?


I don't think so. 
[sys.maxsize](https://docs.python.org/3/library/platform.html#cross-platform) 
indicates both to be 64-bit.

--
https://mail.python.org/mailman/listinfo/python-list

Program uses twice as much memory in Python 3.6 than in Python 3.5

[Jan Gosmann]

Hi,

I have a program which uses twice as much memory when I run it in Python 
3.6 than when I run it in Python 3.5 (about 60GB instead of 30GB). I 
tried to find the reason for that, but the cumulated size (measured with 
sys.getsizeof) of all objects returned by gc.get_objects accumulates 
only to about 17GB in both cases. The program also uses NumPy and I 
tried tracking allocations with the NumPy allocation tracker in the 
relevant part of the program, but again the number of allocations are 
almost identical and the reported maximum memory usage perfectly agrees 
(it is about 18GB).


Any ideas where this significant increase in memory consumption could 
come from? Or any ideas how to further debug this?


Looking at the changelog it seems that the only change in Python 3.6 
affecting memory usage is the new dict implementation which is supposed 
to be more memory efficient. In fact, this is what I find if I run the 
program with a smaller test cases. There the memory consumption is less 
with Python 3.6.


Cheers, Jan
--
https://mail.python.org/mailman/listinfo/python-list

[issue24875] pyvenv doesn´t install PIP inside a new venv with --system-site-package

[Jan Gosmann]

Changes by Jan Gosmann <jan+pythontrac...@hyper-world.de>:


--
nosy: +jgosmann

___
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue24875>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21318] sdist fails with symbolic links do non-existing files

[Jan Gosmann]

New submission from Jan Gosmann:

If there is a symbolic link to a non-existing file anywhere in the source tree 
python setup.py sdist fails with an output like the following:

running sdist
running check
warning: check: missing required meta-data: url

error: abc: No such file or directory

Pruning the problematic path with a MANIFEST.in file does not help.

I could locate the culprit in filelist.py 
http://hg.python.org/cpython/file/c82dcad83438/Lib/distutils/filelist.py in 
line 267:

stat = os.stat(fullname)

fails for symlinks to non-existing files. Maybe os.lstat should be used? Or it 
should be checked before os.stat if it is a symlink to a nonexisting file?

In case you wonder why I have links to non-existing files in my source tree: 
Those can be left behind by automated tests I'm using and are not supposed to 
be part of the source (or any other) distribution. But as I said, the error is 
not prevented by excluding them with a MANIFEST.in file. My current workaround 
is to delete all the leftovers from the test as first thing in setup.py.

--
components: Distutils
messages: 216930
nosy: dstufft, eric.araujo, jgosmann
priority: normal
severity: normal
status: open
title: sdist fails with symbolic links do non-existing files
type: behavior
versions: Python 2.7, Python 3.3

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue21318
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17396] modulefinder fails if module contains syntax error

[Jan Gosmann]

Jan Gosmann added the comment:

Here is an updated patch, also containing a test.

--
Added file: http://bugs.python.org/file30237/fix-handling-of-syntax-errors.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17396
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17396] modulefinder fails if module contains syntax error

[Jan Gosmann]

Jan Gosmann added the comment:

It's based on the default branch becoming 3.4.

--
versions: +Python 3.4 -Python 2.7

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17396
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17396] modulefinder fails if module contains syntax error

[Jan Gosmann]

Jan Gosmann added the comment:

Could you point me to some documentation on how to add a test? I have not been 
involved in Python development so far.

--

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17396
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue17396] modulefinder fails if module contains syntax error

[Jan Gosmann]

New submission from Jan Gosmann:

Running modulefinder.ModuleFinder.run_script on a script which has an import 
statement with a module containing a syntax error will raise a SyntaxError 
exception. I think, modulefinder should instead continue its analysis and add 
the module to badmodules. Especially, as there are valid reasons for importing 
modules with syntax errors like in the following code snippet

if not python3:
from .exec_py2 import exec_
else:
from .exec_py3 import exec_

I attached a patch which changes the code to catch potential SyntaxError 
exceptions and change them to an ImportError.

--
components: Library (Lib)
files: fix-handling-of-syntax-errors.diff
keywords: patch
messages: 183953
nosy: jgosmann
priority: normal
severity: normal
status: open
title: modulefinder fails if module contains syntax error
versions: Python 2.7
Added file: http://bugs.python.org/file29376/fix-handling-of-syntax-errors.diff

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue17396
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com