Hi Guillaume, Thank you for replying so quickly. I think i'm misunderstanding what file storage does. My goal is to have some objects stored in memory (malloc) and others stored on disk (file)
>From what you're saying file method still uses resident memory for each >object, so with my configuration (20G malloc, 75G file) i would need 95G RAM >if all storage is used? (without swap being used) If this is the case i'm curious what the use cases are for file vs only malloc. On Jul 10, 2017, at 1:55 PM, Guillaume Quintard <[email protected]<mailto:[email protected]>> wrote: Hi Charles, So, if I'm reading this right, there's no discrepancy. Varnish will malloc the full storage (malloc), and will mmap a file the size of the full storage (file). So even though the storage is not used, it's allocated. Does it make sense, or did I miss something? -- Guillaume Quintard On Mon, Jul 10, 2017 at 1:23 PM, Bender, Charles <[email protected]<mailto:[email protected]>> wrote: Hi, There is a large discrepancy between Varnish resident memory reported by top vs reported by varnishstat. Varnish is configured with both malloc and file storage; 20G malloc and 75G file storage. These are the startup parameters- VARNISH_STORAGE="memcache=malloc,20G -s filecache=file,/mnt/xvdf1/varnish/varnish_storage.bin,75G" After running for a few days top is reporting more than twice amount of memory used for varnishd process than varnishstat. >From top- PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1083 varnish 20 0 82.235g 0.014t 7.765g S 75.7 51.4 2159:07 varnishd >From varnishstat- SMA.memcache.g_bytes 5.20G 39.65K . 5.20G 5.20G 5.20G SMA.memcache.g_space 14.80G -39.65K . 14.80G 14.80G 14.80G SMF.filecache.g_bytes 7.78G 27.97K . 7.78G 7.78G 7.78G SMF.filecache.g_space 67.22G -27.97K . 67.22G 67.22G 67.22G This is the relevant part of the VCL regarding storage backend selection- sub vcl_backend_response { # define separate cache storage groups if (bereq.http.host ~ "^(encore|thereal|static)\.(beachcamera|buydig)\.com") { set beresp.storage_hint = "filecache"; set beresp.http.X-Cache-Storage = "disk"; } elsif (bereq.url ~ "(?i)\.(jpg|jpeg|gif|ico|pdf|swf|png|zip)") { set beresp.storage_hint = "filecache"; set beresp.http.X-Cache-Storage = "disk"; } elsif (bereq.url ~ "(?i)product\-image\.aspx") { set beresp.storage_hint = "filecache"; set beresp.http.X-Cache-Storage = "disk"; } else { set beresp.storage_hint = "memcache"; set beresp.http.X-Cache-Storage = "memory"; } Can post entire VCL if needed. Would think that since varnishstat reports 5.20G RAM used the resident memory should be around 6-7G, 14G seems excessively high. File storage should use minimal resident memory, correct? Varnish was installed from Varnish Cache 4.1 repo. No VMODs loaded except std and directors. Using latest 4.1.7 Anything else you need please let me know. _______________________________________________ varnish-misc mailing list [email protected]<mailto:[email protected]> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
