Re: What is heating the memory here? hashlib?

2016-02-15 Thread Paulo da Silva
Às 08:12 de 15-02-2016, Johannes Bauer escreveu: > On 15.02.2016 03:21, Paulo da Silva wrote: > >> So far I tried the program twice and it ran perfectly. > > I think you measured your RAM consumption wrong. > > Linux uses all free RAM as HDD cache. That's what is used in "buffers". > That is, it

Re: What is heating the memory here? hashlib?

2016-02-15 Thread Johannes Bauer
On 15.02.2016 03:21, Paulo da Silva wrote: > So far I tried the program twice and it ran perfectly. I think you measured your RAM consumption wrong. Linux uses all free RAM as HDD cache. That's what is used in "buffers". That is, it's not "free", but it would be free if any process would sbrk().

Re: What is heating the memory here? hashlib?

2016-02-15 Thread Paulo da Silva
Às 02:21 de 14-02-2016, Steven D'Aprano escreveu: > On Sun, 14 Feb 2016 06:29 am, Paulo da Silva wrote: ... Thanks Steven for your advices. This is a small script to solve a specific problem. It will be used in future to solve other similar problems probably with small changes. When I found it eat

Re: What is heating the memory here? hashlib?

2016-02-14 Thread Paulo da Silva
Às 09:49 de 14-02-2016, INADA Naoki escreveu: > tracemalloc module may help you to investigate leaks. > 2016/02/14 午後4:05 "Paulo da Silva" : > Thanks. I didn't know it! Paulo -- https://mail.python.org/mailman/listinfo/python-list

Re: What is heating the memory here? hashlib?

2016-02-14 Thread Paulo da Silva
Às 07:04 de 14-02-2016, Paulo da Silva escreveu: > I was unable to reproduce the situation using a simple program just > walking through all files>4K, with or without the seek, and computing > their shasums. > Only some fluctuations of about 500MB in memory consumption. Today I gave another try to

Re: What is heating the memory here? hashlib?

2016-02-14 Thread INADA Naoki
tracemalloc module may help you to investigate leaks. 2016/02/14 午後4:05 "Paulo da Silva" : > I was unable to reproduce the situation using a simple program just > walking through all files>4K, with or without the seek, and computing > their shasums. > Only some fluctuations of about 500MB in memor

Re: What is heating the memory here? hashlib?

2016-02-13 Thread Paulo da Silva
I was unable to reproduce the situation using a simple program just walking through all files>4K, with or without the seek, and computing their shasums. Only some fluctuations of about 500MB in memory consumption. I'll look at this when I get more time, taking in consideration the suggestions here

Re: What is heating the memory here? hashlib?

2016-02-13 Thread Steven D'Aprano
On Sun, 14 Feb 2016 06:29 am, Paulo da Silva wrote: > Hello all. > > I'm running in a very strange (for me at least) problem. > > def getHash(self): > bfsz=File.blksz > h=hashlib.sha256() > hu=h.update > with open(self.getPath(),'rb') as f: > f.seek(File.hdrsz)# Skip header > b=f.read(bfsz)

Re: What is heating the memory here? hashlib?

2016-02-13 Thread Chris Angelico
On Sun, Feb 14, 2016 at 12:44 PM, Paulo da Silva wrote: >> What happens if, after hashing each file (and returning from this >> function), you call gc.collect()? If that reduces your RAM usage, you >> have reference cycles somewhere. >> > I have used gc and del. No luck. > > The most probable caus

Re: What is heating the memory here? hashlib?

2016-02-13 Thread Paulo da Silva
Às 22:45 de 13-02-2016, Chris Angelico escreveu: > On Sun, Feb 14, 2016 at 9:26 AM, Paulo da Silva > wrote: >> I meant eating! :-) > > Heh, "heating" works too - the more you use memory, the more it heats up :) :-) It is heating my head! ... > > What happens if, after hashing each file (and ret

Re: What is heating the memory here? hashlib?

2016-02-13 Thread Chris Angelico
On Sun, Feb 14, 2016 at 9:26 AM, Paulo da Silva wrote: > I meant eating! :-) Heh, "heating" works too - the more you use memory, the more it heats up :) I'm assuming this is inside "class File:" and you have class members for your constants like header size? There's no context for the name "File

Re: What is heating the memory here? hashlib?

2016-02-13 Thread Paulo da Silva
I meant eating! :-) -- https://mail.python.org/mailman/listinfo/python-list

What is heating the memory here? hashlib?

2016-02-13 Thread Paulo da Silva
Hello all. I'm running in a very strange (for me at least) problem. def getHash(self): bfsz=File.blksz h=hashlib.sha256() hu=h.update with open(self.getPath(),'rb') as f: f.seek(File.hdrsz) # Skip