Re: [racket-users] Re: Memory usage of (port->lines ...) is extreamly high

[jackh...@gmail.com]

I'm also guessing the jump from 600MB to 3GB is related to encodings. The 
file is probably UTF8/ASCII, and racket strings are a different encoding. I 
think they're one of the 32-bit encodings? So for ASCII text that alone 
would be a factor of four increase in memory usage.

On Thursday, September 24, 2020 at 5:55:48 AM UTC-7 Sam Tobin-Hochstadt 
wrote:

> port->lines produces a list with all the lines in it. That list is what 
> uses all the memory. Using in-lines avoids producing the whole list at 
> once. 
>
> Sam
>
> On Thu, Sep 24, 2020, 8:53 AM Hong Yang  wrote:
>
>> Thanks Laurent, I tried (in-lines...), and yes, it's memory-efficient, 
>> but I still curious and concern why (port->lines ...) takes so many memory.
>>
>> Best regards
>> Hong
>>
>> On Thursday, September 24, 2020 at 6:55:10 PM UTC+8 laurent...@gmail.com 
>> wrote:
>>
>>> Quick comment: of you don't need to load the whole file but want to 
>>> parse it line by line, use `in-lines` which is memory-efficient.
>>>
>>> On Thu, Sep 24, 2020, 11:46 Hong Yang  wrote:
>>>
 Update with memory dump log attached.

 1. With out (set! input empty), call (collect-garbage) doesn't help
 2. Call (set! input empty) and (collect-garbage), memory reduce 
 dramaticly.

 ; 214M(VIRT)/101M(RSS) without open any file
 (let loop()
   (sleep 5)  ; Waiting here so that I can check it via top/ps
   (dump-memory-stats)
   (collect-garbage)
   (set! input empty)  ; Even not help with this line, it works after 
 called (collect-garbage) explicity.
   (loop))

 On Thursday, September 24, 2020 at 6:14:55 PM UTC+8 Hong Yang wrote:

> Hi Racketer
>
> I'm trying to load a log file which size is 600MB, then I found the 
> program exhausted 3 GB resident memory just for load all the content of 
> it 
> via (port->lines...). I do have enough RAM but it looks like some thing 
> went wrong here, and I do need to load them all into RAM for my use case 
> so 
> (read-line ...) doesn't help me.
>
> Any comment would be preciated.
>
> Here is may programe:
>
> #!/usr/bin/racket
> #lang racket
>
> ; Load input as list of lines
> (define (input-load-lines file-name)
>   (if (file-exists? file-name)
>   (let* ([input (open-input-file file-name)]
>  [lines (port->lines input)])
> (close-input-port input)
> lines)
>   empty))
>
> ; Racket 7.8, compile from source code, (none cs mode)
> ; 100M of log requires 0.5G runtime memory
> ; 300M of log requires 1.5G runtime memory
> ; 600M of log requires 3.0G runtime memory
> ;
> ; Reference
> ;   racket/collects/racket/port.rkt :106
> ;   racket/collects/racket/private/portlines.rkt :11
>
> (define input (input-load-lines "main.log"))
>
> ; 214M(VIRT)/101M(RSS) without open any file
> (let loop()
>   (sleep 5) ; Waiting here so that I can check it via top/ps
>   (set! input empty) ; event not help with this line
>   (loop))
>
> Thanks
> Hong
>
 -- 
 You received this message because you are subscribed to the Google 
 Groups "Racket Users" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to racket-users...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/racket-users/07c41b96-87ba-473a-ad0e-0cec71dc4024n%40googlegroups.com
  
 
 .

>>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Racket Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to racket-users...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/racket-users/f761c588-da7e-43d4-9b05-b917f64f4b52n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/4b11510f-fb14-496b-b7be-b52abfa37982n%40googlegroups.com.

Re: [racket-users] Re: Memory usage of (port->lines ...) is extreamly high

[Sam Tobin-Hochstadt]

port->lines produces a list with all the lines in it. That list is what
uses all the memory. Using in-lines avoids producing the whole list at
once.

Sam

On Thu, Sep 24, 2020, 8:53 AM Hong Yang  wrote:

> Thanks Laurent, I tried (in-lines...), and yes, it's memory-efficient, but
> I still curious and concern why (port->lines ...) takes so many memory.
>
> Best regards
> Hong
>
> On Thursday, September 24, 2020 at 6:55:10 PM UTC+8 laurent...@gmail.com
> wrote:
>
>> Quick comment: of you don't need to load the whole file but want to parse
>> it line by line, use `in-lines` which is memory-efficient.
>>
>> On Thu, Sep 24, 2020, 11:46 Hong Yang  wrote:
>>
>>> Update with memory dump log attached.
>>>
>>> 1. With out (set! input empty), call (collect-garbage) doesn't help
>>> 2. Call (set! input empty) and (collect-garbage), memory reduce
>>> dramaticly.
>>>
>>> ; 214M(VIRT)/101M(RSS) without open any file
>>> (let loop()
>>>   (sleep 5)  ; Waiting here so that I can check it via top/ps
>>>   (dump-memory-stats)
>>>   (collect-garbage)
>>>   (set! input empty)  ; Even not help with this line, it works after
>>> called (collect-garbage) explicity.
>>>   (loop))
>>>
>>> On Thursday, September 24, 2020 at 6:14:55 PM UTC+8 Hong Yang wrote:
>>>
 Hi Racketer

 I'm trying to load a log file which size is 600MB, then I found the
 program exhausted 3 GB resident memory just for load all the content of it
 via (port->lines...). I do have enough RAM but it looks like some thing
 went wrong here, and I do need to load them all into RAM for my use case so
 (read-line ...) doesn't help me.

 Any comment would be preciated.

 Here is may programe:

 #!/usr/bin/racket
 #lang racket

 ; Load input as list of lines
 (define (input-load-lines file-name)
   (if (file-exists? file-name)
   (let* ([input (open-input-file file-name)]
  [lines (port->lines input)])
 (close-input-port input)
 lines)
   empty))

 ; Racket 7.8, compile from source code, (none cs mode)
 ; 100M of log requires 0.5G runtime memory
 ; 300M of log requires 1.5G runtime memory
 ; 600M of log requires 3.0G runtime memory
 ;
 ; Reference
 ;   racket/collects/racket/port.rkt :106
 ;   racket/collects/racket/private/portlines.rkt :11

 (define input (input-load-lines "main.log"))

 ; 214M(VIRT)/101M(RSS) without open any file
 (let loop()
   (sleep 5) ; Waiting here so that I can check it via top/ps
   (set! input empty) ; event not help with this line
   (loop))

 Thanks
 Hong

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Racket Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to racket-users...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/racket-users/07c41b96-87ba-473a-ad0e-0cec71dc4024n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/f761c588-da7e-43d4-9b05-b917f64f4b52n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CAK%3DHD%2Bb1-goK4tz0Yh8axfZdvwq%3DzNCFfva8zu5Sj46%3Dri6Yjw%40mail.gmail.com.

Re: [racket-users] Re: Memory usage of (port->lines ...) is extreamly high

[Hong Yang]

Thanks Laurent, I tried (in-lines...), and yes, it's memory-efficient, but 
I still curious and concern why (port->lines ...) takes so many memory.

Best regards
Hong

On Thursday, September 24, 2020 at 6:55:10 PM UTC+8 laurent...@gmail.com 
wrote:

> Quick comment: of you don't need to load the whole file but want to parse 
> it line by line, use `in-lines` which is memory-efficient.
>
> On Thu, Sep 24, 2020, 11:46 Hong Yang  wrote:
>
>> Update with memory dump log attached.
>>
>> 1. With out (set! input empty), call (collect-garbage) doesn't help
>> 2. Call (set! input empty) and (collect-garbage), memory reduce 
>> dramaticly.
>>
>> ; 214M(VIRT)/101M(RSS) without open any file
>> (let loop()
>>   (sleep 5)  ; Waiting here so that I can check it via top/ps
>>   (dump-memory-stats)
>>   (collect-garbage)
>>   (set! input empty)  ; Even not help with this line, it works after 
>> called (collect-garbage) explicity.
>>   (loop))
>>
>> On Thursday, September 24, 2020 at 6:14:55 PM UTC+8 Hong Yang wrote:
>>
>>> Hi Racketer
>>>
>>> I'm trying to load a log file which size is 600MB, then I found the 
>>> program exhausted 3 GB resident memory just for load all the content of it 
>>> via (port->lines...). I do have enough RAM but it looks like some thing 
>>> went wrong here, and I do need to load them all into RAM for my use case so 
>>> (read-line ...) doesn't help me.
>>>
>>> Any comment would be preciated.
>>>
>>> Here is may programe:
>>>
>>> #!/usr/bin/racket
>>> #lang racket
>>>
>>> ; Load input as list of lines
>>> (define (input-load-lines file-name)
>>>   (if (file-exists? file-name)
>>>   (let* ([input (open-input-file file-name)]
>>>  [lines (port->lines input)])
>>> (close-input-port input)
>>> lines)
>>>   empty))
>>>
>>> ; Racket 7.8, compile from source code, (none cs mode)
>>> ; 100M of log requires 0.5G runtime memory
>>> ; 300M of log requires 1.5G runtime memory
>>> ; 600M of log requires 3.0G runtime memory
>>> ;
>>> ; Reference
>>> ;   racket/collects/racket/port.rkt :106
>>> ;   racket/collects/racket/private/portlines.rkt :11
>>>
>>> (define input (input-load-lines "main.log"))
>>>
>>> ; 214M(VIRT)/101M(RSS) without open any file
>>> (let loop()
>>>   (sleep 5) ; Waiting here so that I can check it via top/ps
>>>   (set! input empty) ; event not help with this line
>>>   (loop))
>>>
>>> Thanks
>>> Hong
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Racket Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to racket-users...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/racket-users/07c41b96-87ba-473a-ad0e-0cec71dc4024n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/f761c588-da7e-43d4-9b05-b917f64f4b52n%40googlegroups.com.

Re: [racket-users] Re: Memory usage of (port->lines ...) is extreamly high

[Laurent]

Quick comment: of you don't need to load the whole file but want to parse
it line by line, use `in-lines` which is memory-efficient.

On Thu, Sep 24, 2020, 11:46 Hong Yang  wrote:

> Update with memory dump log attached.
>
> 1. With out (set! input empty), call (collect-garbage) doesn't help
> 2. Call (set! input empty) and (collect-garbage), memory reduce dramaticly.
>
> ; 214M(VIRT)/101M(RSS) without open any file
> (let loop()
>   (sleep 5)  ; Waiting here so that I can check it via top/ps
>   (dump-memory-stats)
>   (collect-garbage)
>   (set! input empty)  ; Even not help with this line, it works after
> called (collect-garbage) explicity.
>   (loop))
>
> On Thursday, September 24, 2020 at 6:14:55 PM UTC+8 Hong Yang wrote:
>
>> Hi Racketer
>>
>> I'm trying to load a log file which size is 600MB, then I found the
>> program exhausted 3 GB resident memory just for load all the content of it
>> via (port->lines...). I do have enough RAM but it looks like some thing
>> went wrong here, and I do need to load them all into RAM for my use case so
>> (read-line ...) doesn't help me.
>>
>> Any comment would be preciated.
>>
>> Here is may programe:
>>
>> #!/usr/bin/racket
>> #lang racket
>>
>> ; Load input as list of lines
>> (define (input-load-lines file-name)
>>   (if (file-exists? file-name)
>>   (let* ([input (open-input-file file-name)]
>>  [lines (port->lines input)])
>> (close-input-port input)
>> lines)
>>   empty))
>>
>> ; Racket 7.8, compile from source code, (none cs mode)
>> ; 100M of log requires 0.5G runtime memory
>> ; 300M of log requires 1.5G runtime memory
>> ; 600M of log requires 3.0G runtime memory
>> ;
>> ; Reference
>> ;   racket/collects/racket/port.rkt :106
>> ;   racket/collects/racket/private/portlines.rkt :11
>>
>> (define input (input-load-lines "main.log"))
>>
>> ; 214M(VIRT)/101M(RSS) without open any file
>> (let loop()
>>   (sleep 5) ; Waiting here so that I can check it via top/ps
>>   (set! input empty) ; event not help with this line
>>   (loop))
>>
>> Thanks
>> Hong
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-users/07c41b96-87ba-473a-ad0e-0cec71dc4024n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/CABNTSaG_X9qSmAvP%3D0SWp2ozF9FePBi38_%2B5wQGaQaaf5N21mg%40mail.gmail.com.

[racket-users] Re: Memory usage of (port->lines ...) is extreamly high

[Hong Yang]

Update with memory dump log attached.

1. With out (set! input empty), call (collect-garbage) doesn't help
2. Call (set! input empty) and (collect-garbage), memory reduce dramaticly.

; 214M(VIRT)/101M(RSS) without open any file
(let loop()
  (sleep 5)  ; Waiting here so that I can check it via top/ps
  (dump-memory-stats)
  (collect-garbage)
  (set! input empty)  ; Even not help with this line, it works after called 
(collect-garbage) explicity.
  (loop))

On Thursday, September 24, 2020 at 6:14:55 PM UTC+8 Hong Yang wrote:

> Hi Racketer
>
> I'm trying to load a log file which size is 600MB, then I found the 
> program exhausted 3 GB resident memory just for load all the content of it 
> via (port->lines...). I do have enough RAM but it looks like some thing 
> went wrong here, and I do need to load them all into RAM for my use case so 
> (read-line ...) doesn't help me.
>
> Any comment would be preciated.
>
> Here is may programe:
>
> #!/usr/bin/racket
> #lang racket
>
> ; Load input as list of lines
> (define (input-load-lines file-name)
>   (if (file-exists? file-name)
>   (let* ([input (open-input-file file-name)]
>  [lines (port->lines input)])
> (close-input-port input)
> lines)
>   empty))
>
> ; Racket 7.8, compile from source code, (none cs mode)
> ; 100M of log requires 0.5G runtime memory
> ; 300M of log requires 1.5G runtime memory
> ; 600M of log requires 3.0G runtime memory
> ;
> ; Reference
> ;   racket/collects/racket/port.rkt :106
> ;   racket/collects/racket/private/portlines.rkt :11
>
> (define input (input-load-lines "main.log"))
>
> ; 214M(VIRT)/101M(RSS) without open any file
> (let loop()
>   (sleep 5) ; Waiting here so that I can check it via top/ps
>   (set! input empty) ; event not help with this line
>   (loop))
>
> Thanks
> Hong
>

-- 
You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/racket-users/07c41b96-87ba-473a-ad0e-0cec71dc4024n%40googlegroups.com.
Begin Dump
Begin Racket3m
   :   3337 106784
:   9088 603240
  :   4798 165528
 :   6005 240200
  :  150961086912
  :744  29760
   :672  16128
 :  7224
:   7251 232032
  :   9774 408728
   : 11352
 : 19608
 :109   4096
 :182   5824
   : 24768
:312  16392
  :762  91440
  :   1083  69312
   :   2800 204656
   :  1 48
   :896  21504
   :  4208
  :657  35264
   :  14350 577272
   :  5240
   :  9432
  : 1019636731640
  :  2 64
  :  3 96
  :2670211   85446752
 : 35   1120
   :   3355 107360
:5317620  170163840
:  1 32
  :  214441922784
   :   1407  33768
  :549 188856
 :276  68448
 :  10061 241464
  :  1   1200
   : 96   3840
:798  63840
:   6938 393864
:   7594 394760
:   4248 269232
   :  132761129824
   :   1732  55424
:169  12168
  :592 181936
:   7047 394632
   :321  17976
 :508  67336
 :  1 48
:  2 64
   :  1 48
   :  1152
  : 30   3832
 : 58   2320
  : 14560
  :  1 48
  : 43   1032
 :  1 48
 :  1 24
  :  1 64
 :219   5256
  :  2 64
:  12494 399808
  :  2192
  : 14   1456
  : 12480
 :  1 32
  :  3 598160
   :  1304
   :  1112
:  1104
:277  22160
:  1 40
:273  24024
  : 94   4512
:  6288
   :  6384
  :  1 24
  :  133571068560
:   1229