Update with memory dump log attached.
1. With out (set! input empty), call (collect-garbage) doesn't help
2. Call (set! input empty) and (collect-garbage), memory reduce dramaticly.
; 214M(VIRT)/101M(RSS) without open any file
(let loop()
(sleep 5) ; Waiting here so that I can check it via top/ps
(dump-memory-stats)
(collect-garbage)
(set! input empty) ; Even not help with this line, it works after called
(collect-garbage) explicity.
(loop))
On Thursday, September 24, 2020 at 6:14:55 PM UTC+8 Hong Yang wrote:
> Hi Racketer
>
> I'm trying to load a log file which size is 600MB, then I found the
> program exhausted 3 GB resident memory just for load all the content of it
> via (port->lines...). I do have enough RAM but it looks like some thing
> went wrong here, and I do need to load them all into RAM for my use case so
> (read-line ...) doesn't help me.
>
> Any comment would be preciated.
>
> Here is may programe:
>
> #!/usr/bin/racket
> #lang racket
>
> ; Load input as list of lines
> (define (input-load-lines file-name)
> (if (file-exists? file-name)
> (let* ([input (open-input-file file-name)]
> [lines (port->lines input)])
> (close-input-port input)
> lines)
> empty))
>
> ; Racket 7.8, compile from source code, (none cs mode)
> ; 100M of log requires 0.5G runtime memory
> ; 300M of log requires 1.5G runtime memory
> ; 600M of log requires 3.0G runtime memory
> ;
> ; Reference
> ; racket/collects/racket/port.rkt :106
> ; racket/collects/racket/private/portlines.rkt :11
>
> (define input (input-load-lines "main.log"))
>
> ; 214M(VIRT)/101M(RSS) without open any file
> (let loop()
> (sleep 5) ; Waiting here so that I can check it via top/ps
> (set! input empty) ; event not help with this line
> (loop))
>
> Thanks
> Hong
>
--
You received this message because you are subscribed to the Google Groups
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to racket-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/racket-users/07c41b96-87ba-473a-ad0e-0cec71dc4024n%40googlegroups.com.
Begin Dump
Begin Racket3m
<variable-code>: 3337 106784
<application-code>: 9088 603240
<unary-application-c: 9686 309952
<binary-application-: 11948 477920
<sequence-code>: 4798 165528
<branch-code>: 6005 240200
<procedure-code>: 15096 1086912
<let-value-code>: 744 29760
<let-void-code>: 672 16128
<letrec-code>: 7 224
<let-one-code>: 7251 232032
<with-continuation-m: 72 2880
<define-values-code>: 9774 408728
<set!-code>: 11 352
<boxenv-code>: 19 608
<begin0-code>: 109 4096
<varref-code>: 182 5824
<apply-values-code>: 24 768
<case-lambda-code>: 312 16392
<inline-variant-code: 850 40800
=BYTECODE-TOTAL=: 79985 3749128
<linklet>: 762 91440
<global-variable-cod: 14471 694608
<instance>: 1083 69312
<procedure>: 2800 204656
<procedure>: 1 48
<procedure>: 896 21504
<procedure>: 4 208
<struct>: 657 35264
<procedure>: 14350 577272
<chaperone>: 5 240
<chaperone>: 9 432
<struct>: 101963 6731640
<bignum-integer>: 2 64
<complex-number>: 3 96
<string>: 2670211 85446752
<byte-string>: 35 1120
<unix-path>: 3355 107360
<pair>: 5317620 170163840
<mutable-pair>: 1 32
<vector>: 21444 1922784
<inspector>: 1407 33768
<input-port>: 549 188856
<output-port>: 276 68448
<box>: 10061 241464
<thread>: 1 1200
<semaphore>: 96 3840
<hash>: 798 63840
<hash>: 6938 393864
<hash>: 7594 394760
<hash>: 4248 269232
<hash-node>: 13276 1129824
<hash-table-bucket>: 1732 55424
<hash>: 169 12168
<runtime-prefix>: 592 181936
<weak-box>: 7047 394632
<ephemeron>: 321 17976
<struct-type>: 508 67336
<env>: 1 48
<parameterization>: 2 64
<will-executor>: 1 48
<custodian>: 1 152
<regexp>: 30 3832
<struct-property>: 58 2320
<chaperone-property>: 14 560
<security-guard>: 1 48
<semaphore-peek>: 43 1032
<channel>: 1 48
<thread-dead-evt>: 1 24
<thread-set>: 1 64
<thread-cell>: 219 5256
<variable-reference>: 2 64
<raw-pair>: 12494 399808
<prompt>: 2 192
<continuation-prompt: 7 224
<continuation-mark-k: 2 48
<logger>: 14 1456
<environment-variabl: 1 24
<ctype>: 12 480
<plumber>: 1 32
<plumber-flush-handl: 2 64
<weak-array>: 3 598160
<overflow-jump>: 1 304
<meta-continuation>: 1 112
<dynamic-wind>: 1 104
<input-fd>: 277 22160
<string-port-data>: 1 40
<pipe>: 273 24024
<param-data>: 94 4512
<finalization>: 6 288
<finalizations>: 6 384
<thread-hop>: 1 24
<internal-parameteri: 2 1040
<native-code>: 13357 1068560
<native-code+case>: 1229 108152
<delay-load-info>: 273 41496
<runstack>: 1 8208
End Racket3m
Generation 0: 2012360 of 33554432 bytes used
Generation 0.5: 0 bytes used
Generation 1 [tagged]: 104446456 bytes used of 105005744 in 6425 pages
Generation 1 [atomic]: 1289723080 bytes used of 1289822824 in 80151 pages
Generation 1 [array]: 2520704 bytes used of 2603136 in 164 pages
Generation 1 [pair]: 170563680 bytes used of 170620288 in 10436 pages
Generation 1 [big]: 5313624 bytes used of 5313624 in 102 pages
Generation 1 [medium]: 258 [1/2064] 1026 [1/8208]
Generation 1 [medium atomic]:
Current memory use: 1574590176
part current use from child places: 0
Peak memory use before a collection: 1572643352
Peak memory use after a collection: 1572577816
Allocated (+reserved) page sizes: 1632059392 (+14254080)
Phantom bytes: 0
Past allocated memory: 2417831608
# of major collections: 5
# of minor collections: 76
# of installed finalizers: 2431
# of traced ephemerons: 0
# of immobile boxes: 0
# of page-modify unprotects: 2307
# of old regions scanned during minor GCs: 1905/3156897
JIT-generated code: 4784128
Marshaled code: 5420576
Begin Help
(dump-memory-stats 'count sym) - return number of instances of type named by
sym
Example: (dump-memory-stats 'count '<pair>)
End Help
End Dump
Begin Dump
Begin Racket3m
<variable-code>: 3337 106784
<application-code>: 9088 603240
<unary-application-c: 9686 309952
<binary-application-: 11948 477920
<sequence-code>: 4798 165528
<branch-code>: 6005 240200
<procedure-code>: 15096 1086912
<let-value-code>: 744 29760
<let-void-code>: 672 16128
<letrec-code>: 7 224
<let-one-code>: 7251 232032
<with-continuation-m: 72 2880
<define-values-code>: 9774 408728
<set!-code>: 11 352
<boxenv-code>: 19 608
<begin0-code>: 109 4096
<varref-code>: 182 5824
<apply-values-code>: 24 768
<case-lambda-code>: 312 16392
<inline-variant-code: 850 40800
=BYTECODE-TOTAL=: 79985 3749128
<linklet>: 762 91440
<global-variable-cod: 14471 694608
<instance>: 1083 69312
<procedure>: 2800 204656
<procedure>: 1 48
<procedure>: 896 21504
<procedure>: 4 208
<struct>: 657 35264
<procedure>: 14351 577312
<chaperone>: 5 240
<chaperone>: 9 432
<struct>: 101963 6731640
<bignum-integer>: 2 64
<complex-number>: 3 96
<string>: 2670211 85446752
<byte-string>: 35 1120
<unix-path>: 3354 107328
<pair>: 2732642 87444544
<mutable-pair>: 1 32
<vector>: 21444 1922784
<inspector>: 1407 33768
<input-port>: 548 188512
<output-port>: 276 68448
<box>: 10061 241464
<thread>: 1 1200
<semaphore>: 96 3840
<hash>: 798 63840
<hash>: 6938 393864
<hash>: 7594 394760
<hash>: 4248 269232
<hash-node>: 13276 1129824
<hash-table-bucket>: 1732 55424
<hash>: 169 12168
<runtime-prefix>: 592 181936
<weak-box>: 7045 394520
<ephemeron>: 321 17976
<struct-type>: 508 67336
<env>: 1 48
<parameterization>: 2 64
<will-executor>: 1 48
<custodian>: 1 152
<regexp>: 30 3832
<struct-property>: 58 2320
<chaperone-property>: 14 560
<security-guard>: 1 48
<semaphore-peek>: 43 1032
<channel>: 1 48
<thread-dead-evt>: 1 24
<thread-set>: 1 64
<thread-cell>: 219 5256
<variable-reference>: 2 64
<raw-pair>: 12493 399776
<prompt>: 2 192
<continuation-prompt: 7 224
<continuation-mark-k: 2 48
<logger>: 14 1456
<environment-variabl: 1 24
<ctype>: 12 480
<plumber>: 1 32
<plumber-flush-handl: 2 64
<weak-array>: 3 598160
<overflow-jump>: 1 304
<meta-continuation>: 1 112
<dynamic-wind>: 1 104
<input-fd>: 276 22080
<string-port-data>: 1 40
<pipe>: 273 24024
<param-data>: 94 4512
<finalization>: 5 240
<finalizations>: 5 320
<thread-hop>: 1 24
<internal-parameteri: 2 1040
<native-code>: 13357 1068560
<native-code+case>: 1229 108152
<delay-load-info>: 273 41496
<runstack>: 1 8208
End Racket3m
Generation 0: 0 of 33554432 bytes used
Generation 0.5: 0 bytes used
Generation 1 [tagged]: 104445816 bytes used of 105005832 in 6425 pages
Generation 1 [atomic]: 1289717960 bytes used of 1289822824 in 80150 pages
Generation 1 [array]: 2520688 bytes used of 2603136 in 164 pages
Generation 1 [pair]: 87844352 bytes used of 87908896 in 5377 pages
Generation 1 [big]: 5313624 bytes used of 5313624 in 102 pages
Generation 1 [medium]: 258 [1/2064] 1026 [1/8208]
Generation 1 [medium atomic]:
Current memory use: 1489852712
part current use from child places: 0
Peak memory use before a collection: 1574590176
Peak memory use after a collection: 1572577816
Allocated (+reserved) page sizes: 1549156352 (+97157120)
Phantom bytes: 0
Past allocated memory: 2419843968
# of major collections: 6
# of minor collections: 76
# of installed finalizers: 2431
# of traced ephemerons: 321
# of immobile boxes: 0
# of page-modify unprotects: 2310
# of old regions scanned during minor GCs: 1905/3156897
JIT-generated code: 4784128
Marshaled code: 5420576
Begin Help
(dump-memory-stats 'count sym) - return number of instances of type named by
sym
Example: (dump-memory-stats 'count '<pair>)
End Help
End Dump
Begin Dump
Begin Racket3m
<variable-code>: 3337 106784
<application-code>: 9088 603240
<unary-application-c: 9686 309952
<binary-application-: 11948 477920
<sequence-code>: 4798 165528
<branch-code>: 6005 240200
<procedure-code>: 15096 1086912
<let-value-code>: 744 29760
<let-void-code>: 672 16128
<letrec-code>: 7 224
<let-one-code>: 7251 232032
<with-continuation-m: 72 2880
<define-values-code>: 9774 408728
<set!-code>: 11 352
<boxenv-code>: 19 608
<begin0-code>: 109 4096
<varref-code>: 182 5824
<apply-values-code>: 24 768
<case-lambda-code>: 312 16392
<inline-variant-code: 850 40800
=BYTECODE-TOTAL=: 79985 3749128
<linklet>: 762 91440
<global-variable-cod: 14471 694608
<instance>: 1083 69312
<procedure>: 2800 204656
<procedure>: 1 48
<procedure>: 896 21504
<procedure>: 4 208
<struct>: 657 35264
<procedure>: 14351 577312
<chaperone>: 5 240
<chaperone>: 9 432
<struct>: 101963 6731640
<bignum-integer>: 2 64
<complex-number>: 3 96
<string>: 1940 62080
<byte-string>: 35 1120
<unix-path>: 3354 107328
<pair>: 64371 2059872
<mutable-pair>: 1 32
<vector>: 21444 1922784
<inspector>: 1407 33768
<input-port>: 548 188512
<output-port>: 276 68448
<box>: 10061 241464
<thread>: 1 1200
<semaphore>: 96 3840
<hash>: 798 63840
<hash>: 6938 393864
<hash>: 7594 394760
<hash>: 4248 269232
<hash-node>: 13276 1129824
<hash-table-bucket>: 1732 55424
<hash>: 169 12168
<runtime-prefix>: 592 181936
<weak-box>: 7045 394520
<ephemeron>: 321 17976
<struct-type>: 508 67336
<env>: 1 48
<parameterization>: 2 64
<will-executor>: 1 48
<custodian>: 1 152
<regexp>: 30 3832
<struct-property>: 58 2320
<chaperone-property>: 14 560
<security-guard>: 1 48
<semaphore-peek>: 43 1032
<channel>: 1 48
<thread-dead-evt>: 1 24
<thread-set>: 1 64
<thread-cell>: 219 5256
<variable-reference>: 2 64
<raw-pair>: 12493 399776
<prompt>: 2 192
<continuation-prompt: 7 224
<continuation-mark-k: 2 48
<logger>: 14 1456
<environment-variabl: 1 24
<ctype>: 12 480
<plumber>: 1 32
<plumber-flush-handl: 2 64
<weak-array>: 3 598160
<overflow-jump>: 1 304
<meta-continuation>: 1 112
<dynamic-wind>: 1 104
<input-fd>: 276 22080
<string-port-data>: 1 40
<pipe>: 273 24024
<param-data>: 94 4512
<finalization>: 5 240
<finalizations>: 5 320
<thread-hop>: 1 24
<internal-parameteri: 2 1040
<native-code>: 13357 1068560
<native-code+case>: 1229 108152
<delay-load-info>: 273 41496
<runstack>: 1 8208
End Racket3m
Generation 0: 0 of 33554432 bytes used
Generation 0.5: 0 bytes used
Generation 1 [tagged]: 19061144 bytes used of 19648392 in 1205 pages
Generation 1 [atomic]: 4944720 bytes used of 5045136 in 333 pages
Generation 1 [array]: 2520688 bytes used of 2603136 in 164 pages
Generation 1 [pair]: 2459680 bytes used of 2535328 in 156 pages
Generation 1 [big]: 5313624 bytes used of 5313624 in 102 pages
Generation 1 [medium]: 258 [1/2064] 1026 [1/8208]
Generation 1 [medium atomic]:
Current memory use: 34310128
part current use from child places: 0
Peak memory use before a collection: 1574590176
Peak memory use after a collection: 1572577816
Allocated (+reserved) page sizes: 70369280 (+107970560)
Phantom bytes: 0
Past allocated memory: 2419843968
# of major collections: 7
# of minor collections: 76
# of installed finalizers: 2431
# of traced ephemerons: 321
# of immobile boxes: 0
# of page-modify unprotects: 2314
# of old regions scanned during minor GCs: 1905/3156897
JIT-generated code: 4784128
Marshaled code: 5420576
Begin Help
(dump-memory-stats 'count sym) - return number of instances of type named by
sym
Example: (dump-memory-stats 'count '<pair>)
End Help
End Dump
Begin Dump
Begin Racket3m
<variable-code>: 3337 106784
<application-code>: 9088 603240
<unary-application-c: 9686 309952
<binary-application-: 11948 477920
<sequence-code>: 4798 165528
<branch-code>: 6005 240200
<procedure-code>: 15096 1086912
<let-value-code>: 744 29760
<let-void-code>: 672 16128
<letrec-code>: 7 224
<let-one-code>: 7251 232032
<with-continuation-m: 72 2880
<define-values-code>: 9774 408728
<set!-code>: 11 352
<boxenv-code>: 19 608
<begin0-code>: 109 4096
<varref-code>: 182 5824
<apply-values-code>: 24 768
<case-lambda-code>: 312 16392
<inline-variant-code: 850 40800
=BYTECODE-TOTAL=: 79985 3749128
<linklet>: 762 91440
<global-variable-cod: 14471 694608
<instance>: 1083 69312
<procedure>: 2800 204656
<procedure>: 1 48
<procedure>: 896 21504
<procedure>: 4 208
<struct>: 657 35264
<procedure>: 14351 577312
<chaperone>: 5 240
<chaperone>: 9 432
<struct>: 101963 6731640
<bignum-integer>: 2 64
<complex-number>: 3 96
<string>: 1940 62080
<byte-string>: 35 1120
<unix-path>: 3354 107328
<pair>: 64371 2059872
<mutable-pair>: 1 32
<vector>: 21444 1922784
<inspector>: 1407 33768
<input-port>: 548 188512
<output-port>: 276 68448
<box>: 10061 241464
<thread>: 1 1200
<semaphore>: 96 3840
<hash>: 798 63840
<hash>: 6938 393864
<hash>: 7594 394760
<hash>: 4248 269232
<hash-node>: 13276 1129824
<hash-table-bucket>: 1732 55424
<hash>: 169 12168
<runtime-prefix>: 592 181936
<weak-box>: 7045 394520
<ephemeron>: 321 17976
<struct-type>: 508 67336
<env>: 1 48
<parameterization>: 2 64
<will-executor>: 1 48
<custodian>: 1 152
<regexp>: 30 3832
<struct-property>: 58 2320
<chaperone-property>: 14 560
<security-guard>: 1 48
<semaphore-peek>: 43 1032
<channel>: 1 48
<thread-dead-evt>: 1 24
<thread-set>: 1 64
<thread-cell>: 219 5256
<variable-reference>: 2 64
<raw-pair>: 12493 399776
<prompt>: 2 192
<continuation-prompt: 7 224
<continuation-mark-k: 2 48
<logger>: 14 1456
<environment-variabl: 1 24
<ctype>: 12 480
<plumber>: 1 32
<plumber-flush-handl: 2 64
<weak-array>: 3 598160
<overflow-jump>: 1 304
<meta-continuation>: 1 112
<dynamic-wind>: 1 104
<input-fd>: 276 22080
<string-port-data>: 1 40
<pipe>: 273 24024
<param-data>: 94 4512
<finalization>: 5 240
<finalizations>: 5 320
<thread-hop>: 1 24
<internal-parameteri: 2 1040
<native-code>: 13357 1068560
<native-code+case>: 1229 108152
<delay-load-info>: 273 41496
<runstack>: 1 8208
End Racket3m
Generation 0: 0 of 33554432 bytes used
Generation 0.5: 0 bytes used
Generation 1 [tagged]: 19061144 bytes used of 19621016 in 1201 pages
Generation 1 [atomic]: 4944720 bytes used of 5034328 in 331 pages
Generation 1 [array]: 2520688 bytes used of 2603136 in 164 pages
Generation 1 [pair]: 2459680 bytes used of 2519008 in 155 pages
Generation 1 [big]: 5313624 bytes used of 5313624 in 102 pages
Generation 1 [medium]: 258 [1/2064] 1026 [1/8208]
Generation 1 [medium atomic]:
Current memory use: 34310128
part current use from child places: 0
Peak memory use before a collection: 1574590176
Peak memory use after a collection: 1572577816
Allocated (+reserved) page sizes: 70254592 (+108085248)
Phantom bytes: 0
Past allocated memory: 2419843968
# of major collections: 8
# of minor collections: 76
# of installed finalizers: 2431
# of traced ephemerons: 321
# of immobile boxes: 0
# of page-modify unprotects: 2318
# of old regions scanned during minor GCs: 1905/3156897
JIT-generated code: 4784128
Marshaled code: 5420576
Begin Help
(dump-memory-stats 'count sym) - return number of instances of type named by
sym
Example: (dump-memory-stats 'count '<pair>)
End Help
End Dump
Begin Dump
Begin Racket3m
<variable-code>: 3337 106784
<application-code>: 9088 603240
<unary-application-c: 9686 309952
<binary-application-: 11948 477920
<sequence-code>: 4798 165528
<branch-code>: 6005 240200
<procedure-code>: 15096 1086912
<let-value-code>: 744 29760
<let-void-code>: 672 16128
<letrec-code>: 7 224
<let-one-code>: 7251 232032
<with-continuation-m: 72 2880
<define-values-code>: 9774 408728
<set!-code>: 11 352
<boxenv-code>: 19 608
<begin0-code>: 109 4096
<varref-code>: 182 5824
<apply-values-code>: 24 768
<case-lambda-code>: 312 16392
<inline-variant-code: 850 40800
=BYTECODE-TOTAL=: 79985 3749128
<linklet>: 762 91440
<global-variable-cod: 14471 694608
<instance>: 1083 69312
<procedure>: 2800 204656
<procedure>: 1 48
<procedure>: 896 21504
<procedure>: 4 208
<struct>: 657 35264
<procedure>: 14351 577312
<chaperone>: 5 240
<chaperone>: 9 432
<struct>: 101963 6731640
<bignum-integer>: 2 64
<complex-number>: 3 96
<string>: 1940 62080
<byte-string>: 35 1120
<unix-path>: 3354 107328
<pair>: 64371 2059872
<mutable-pair>: 1 32
<vector>: 21444 1922784
<inspector>: 1407 33768
<input-port>: 548 188512
<output-port>: 276 68448
<box>: 10061 241464
<thread>: 1 1200
<semaphore>: 96 3840
<hash>: 798 63840
<hash>: 6938 393864
<hash>: 7594 394760
<hash>: 4248 269232
<hash-node>: 13276 1129824
<hash-table-bucket>: 1732 55424
<hash>: 169 12168
<runtime-prefix>: 592 181936
<weak-box>: 7045 394520
<ephemeron>: 321 17976
<struct-type>: 508 67336
<env>: 1 48
<parameterization>: 2 64
<will-executor>: 1 48
<custodian>: 1 152
<regexp>: 30 3832
<struct-property>: 58 2320
<chaperone-property>: 14 560
<security-guard>: 1 48
<semaphore-peek>: 43 1032
<channel>: 1 48
<thread-dead-evt>: 1 24
<thread-set>: 1 64
<thread-cell>: 219 5256
<variable-reference>: 2 64
<raw-pair>: 12493 399776
<prompt>: 2 192
<continuation-prompt: 7 224
<continuation-mark-k: 2 48
<logger>: 14 1456
<environment-variabl: 1 24
<ctype>: 12 480
<plumber>: 1 32
<plumber-flush-handl: 2 64
<weak-array>: 3 598160
<overflow-jump>: 1 304
<meta-continuation>: 1 112
<dynamic-wind>: 1 104
<input-fd>: 276 22080
<string-port-data>: 1 40
<pipe>: 273 24024
<param-data>: 94 4512
<finalization>: 5 240
<finalizations>: 5 320
<thread-hop>: 1 24
<internal-parameteri: 2 1040
<native-code>: 13357 1068560
<native-code+case>: 1229 108152
<delay-load-info>: 273 41496
<runstack>: 1 8208
End Racket3m
Generation 0: 0 of 33554432 bytes used
Generation 0.5: 0 bytes used
Generation 1 [tagged]: 19061144 bytes used of 19585000 in 1199 pages
Generation 1 [atomic]: 4944720 bytes used of 5033480 in 331 pages
Generation 1 [array]: 2520688 bytes used of 2602640 in 162 pages
Generation 1 [pair]: 2459680 bytes used of 2515424 in 154 pages
Generation 1 [big]: 5313624 bytes used of 5313624 in 102 pages
Generation 1 [medium]: 258 [1/2064] 1026 [1/8208]
Generation 1 [medium atomic]:
Current memory use: 34310128
part current use from child places: 0
Peak memory use before a collection: 1574590176
Peak memory use after a collection: 1572577816
Allocated (+reserved) page sizes: 70172672 (+74612736)
Phantom bytes: 0
Past allocated memory: 2419843968
# of major collections: 9
# of minor collections: 76
# of installed finalizers: 2431
# of traced ephemerons: 321
# of immobile boxes: 0
# of page-modify unprotects: 2322
# of old regions scanned during minor GCs: 1905/3156897
JIT-generated code: 4784128
Marshaled code: 5420576
Begin Help
(dump-memory-stats 'count sym) - return number of instances of type named by
sym
Example: (dump-memory-stats 'count '<pair>)
End Help
End Dump
