Re: Warm restart setup for dummies
I got it to work! With your great help it wasn't that bad, or maybe I'm a better sysadmin than I'm giving myself credit for. ;) There were two steps, really. After creating the fstab line, I typed sudo systemctl edit memcached.service. I entered this and saved: [Service] KillSignal=SIGUSR1 Then it's a matter of: 0. sudo systemctl daemon-reload 1. stopping and disabling the systemd services for the webserver 2. Stopping and disabling memcached 3. copying the memory_file and memory_file.meta to a safe location on the hard disk 4. rebooting 5. copying the files back to /tmpfs_mount_memcached 6. Starting and enabling memcached with systemctl 7. Starting and enabling the webserver All of these steps I was able to easily automate with Ansible. Really happy about this! Thanks again! torsdag 11. juni 2020 08.53.19 UTC+2 skrev Dormando følgende: > > Absolutely. That's exactly the workflow it's designed for, we just haven't > updated any of the systemd scripts to be more friendly for it. > > Also a caveat; there _was_ a bug fixed relatively recently with the > restart code. I don't know if ubuntu backports these. If you use large > objects (> 512k) there's a chance restart won't work sometimes. Worst case > you can probably file a bug with them to backport the patch or upgrade > memcached. > > Good luck! > > On Wed, 10 Jun 2020, Even Onsager wrote: > > > That's extremely helpful, thank you so much for this! I will look into > it and test on my staging server. I don't think systemd has ever killed or > restarted the process apart from once before I upgraded the RAM, so I'm not > too worried about the daily usage. But even systemd supports custom kill > signals, so it should be possible to set this up? > > > > Anyway, it's the reboots I'm trying to get to work. I never upgrade apt > packages or reboot directly, only with Ansible after kernel upgrades or > similar, so I should be able to disable the systemd services (should > probably temporarily disable the puma webserver service too) and automate a > copy to disk task before the reboot takes place. A good thing with Ansible > is that it can automate reboots and continue with more tasks after reboot > is complete, so it should be ideal for this scenario. I will post back if I > can get it to work, should be interesting for more than me. :) > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > Groups "memcached" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to memc...@googlegroups.com . > > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/5ec346ab-6977-4996-b573-9d07dd0d4084o%40googlegroups.com. > > > > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/7a2b1b42-cb51-48f6-b00d-f0a4bbb1be77o%40googlegroups.com.
Re: Warm restart setup for dummies
Absolutely. That's exactly the workflow it's designed for, we just haven't updated any of the systemd scripts to be more friendly for it. Also a caveat; there _was_ a bug fixed relatively recently with the restart code. I don't know if ubuntu backports these. If you use large objects (> 512k) there's a chance restart won't work sometimes. Worst case you can probably file a bug with them to backport the patch or upgrade memcached. Good luck! On Wed, 10 Jun 2020, Even Onsager wrote: > That's extremely helpful, thank you so much for this! I will look into it and > test on my staging server. I don't think systemd has ever killed or restarted > the process apart from once before I upgraded the RAM, so I'm not too worried > about the daily usage. But even systemd supports custom kill signals, so it > should be possible to set this up? > > Anyway, it's the reboots I'm trying to get to work. I never upgrade apt > packages or reboot directly, only with Ansible after kernel upgrades or > similar, so I should be able to disable the systemd services (should probably > temporarily disable the puma webserver service too) and automate a copy to > disk task before the reboot takes place. A good thing with Ansible is that it > can automate reboots and continue with more tasks after reboot is complete, > so it should be ideal for this scenario. I will post back if I can get it to > work, should be interesting for more than me. :) > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/5ec346ab-6977-4996-b573-9d07dd0d4084o%40googlegroups.com. > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.2006102350440.27659%40dskull.
Re: Warm restart setup for dummies
That's extremely helpful, thank you so much for this! I will look into it and test on my staging server. I don't think systemd has ever killed or restarted the process apart from once before I upgraded the RAM, so I'm not too worried about the daily usage. But even systemd supports custom kill signals, so it should be possible to set this up? Anyway, it's the reboots I'm trying to get to work. I never upgrade apt packages or reboot directly, only with Ansible after kernel upgrades or similar, so I should be able to disable the systemd services (should probably temporarily disable the puma webserver service too) and automate a copy to disk task before the reboot takes place. A good thing with Ansible is that it can automate reboots and continue with more tasks after reboot is complete, so it should be ideal for this scenario. I will post back if I can get it to work, should be interesting for more than me. :) -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/5ec346ab-6977-4996-b573-9d07dd0d4084o%40googlegroups.com.
Re: Warm restart setup for dummies
Hey, I might have to look at how ubuntu's install works.. it might not be set up for this. These are the basic steps for a restart: 1) set up memcached as you did, tmpfs/etc. 2) when you want to stop gracefully, issue a `kill -SIGUSR1 $(pidof memcached)` (kill is the command to send signals to a process) 3) start memcached again with the same options, and it will recover its data. This will _not_ survive reboots. This will survive software upgrades, which ubuntu isn't going to do anyway :) To survive reboots you need a few more steps: 1) once memcached has stopped, copy the files created in /tmpfs_mount_memcached/ to an actual harddrive somewhere. 2) reboot. 3) copy the datafiles back in place. 4) start memcached again. For this to be reliable you probably don't want ubuntu to automatically manage the process (disable auto start/restart in systemd). You might want to engage some community/chat/something for some systems admnistration help to get this going. Sounds like you're a bit over your head :( good luck, -Dormando On Wed, 10 Jun 2020, Even Onsager wrote: > My site runs on one webserver and we rely heavily on memcached to make it > snappy, to the extent that a reboot will make the site unresponsive for hours. > So imagine my joy when I saw the warm restart addition, and the fact that > Ubuntu Server 20.04 LTS has a new enough version in its repo. > > But the wiki left me scratching my head. This is what I have: > > - The standard apt package for Ubuntu 20.04 (version 1.5.22) > - `-e /tmpfs_mount_memcached/memory_file` in memcached.conf > - `-m 920` in memcached.conf > - `tmpfs /tmpfs_mount_memcached tmpfs nodev,nosuid,size=930M 0` in /etc/fstab > (generated by Ansible's mount module) > > No type of restart (neither of the systemd service nor the server itself) > seems to work. After restarting the size of the cache store is 0 and all pages > take forever to load. But is it supposed to work like this with tmpfs mounts? > I thought tmpfs wasn't meant to survive reboots? Am I misreading the wiki? > > I'm obviously in way over my head here (I don't even really know what a > SIGUSR1 is), so I'd really appreciate some help as to what I'm missing! > > And thanks for memcached - it's served my site well for years now, and after > upgrading to Rails 5 with better caching, we're using it more and more. :) > > -- > > --- > You received this message because you are subscribed to the Google Groups > "memcached" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to memcached+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/memcached/afb47849-1915-4183-9765-457e7e4bc153o%40googlegroups.com. > > -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/alpine.DEB.2.21.2006101506060.27659%40dskull.
Warm restart setup for dummies
My site runs on one webserver and we rely heavily on memcached to make it snappy, to the extent that a reboot will make the site unresponsive for hours. So imagine my joy when I saw the warm restart addition, and the fact that Ubuntu Server 20.04 LTS has a new enough version in its repo. But the wiki left me scratching my head. This is what I have: - The standard apt package for Ubuntu 20.04 (version 1.5.22) - `-e /tmpfs_mount_memcached/memory_file` in memcached.conf - `-m 920` in memcached.conf - `tmpfs /tmpfs_mount_memcached tmpfs nodev,nosuid,size=930M 0` in /etc/fstab (generated by Ansible's mount module) No type of restart (neither of the systemd service nor the server itself) seems to work. After restarting the size of the cache store is 0 and all pages take forever to load. But is it supposed to work like this with tmpfs mounts? I thought tmpfs wasn't meant to survive reboots? Am I misreading the wiki? I'm obviously in way over my head here (I don't even really know what a SIGUSR1 is), so I'd really appreciate some help as to what I'm missing! And thanks for memcached - it's served my site well for years now, and after upgrading to Rails 5 with better caching, we're using it more and more. :) -- --- You received this message because you are subscribed to the Google Groups "memcached" group. To unsubscribe from this group and stop receiving emails from it, send an email to memcached+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/afb47849-1915-4183-9765-457e7e4bc153o%40googlegroups.com.
