On Thursday, January 16, 2020 09:15 GMT, Dan van der Ster
wrote:
> Hi Nick,
>
> We saw the exact same problem yesterday after a network outage -- a few of
> our down OSDs were stuck down until we restarted their processes.
>
> -- Dan
>
>
> On Wed, Jan 15, 2020 at 3:37 PM Nick Fisk wrote:
>
> > Hi All,
> >
> > Running 14.2.5, currently experiencing some network blips isolated to a
> > single rack which is under investigation. However, it appears following a
> > network blip, random OSD's in unaffected racks are sometimes not recovering
> > from the incident and are left running running in a zombie state. The OSD's
> > appear to be running from a process perspective, but the cluster thinks
> > they are down and will not rejoin the cluster until the OSD process is
> > restarted, which incidentally takes a lot longer than usual (systemctl
> > command takes a couple of minutes to complete).
> >
> > If the OSD is left in this state, CPU and memory usage of the process
> > appears to climb, but never rejoins, at least for several hours that I have
> > left them. Not exactly sure what the OSD is trying to do during this
> > period. There's nothing in the logs during this hung state to indicate that
> > anything is happening, but I will try and inject more verbose logging next
> > time it occurs.
> >
> > Not sure if anybody has come across this before or any ideas? In the past
> > as long as OSD's have been running they have always re-joint following any
> > network issues.
> >
> > Nick
> >
> > Sample from OSD and cluster logs below. Blip happened at 12:06, I
> > restarted OSD at 12:26
> >
> > OSD Logs from OSD that hung (Note this OSD was not directly affected by
> > network outage)
> > 2020-01-15 12:06:32.234 7f41a1023700 -1 osd.43 2342991 heartbeat_check: no
> > reply from [*:*:*:5::14]:6838 osd.71 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:33.194 7f41a1023700 -1 osd.43 2342991 heartbeat_check: no
> > reply from [*:*:*:5::13]:6854 osd.49 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:33.194 7f41a1023700 -1 osd.43 2342991 heartbeat_check: no
> > reply from [*:*:*:5::13]:6834 osd.51 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:33.194 7f41a1023700 -1 osd.43 2342991 heartbeat_check: no
> > reply from [*:*:*:5::13]:6862 osd.52 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:33.194 7f41a1023700 -1 osd.43 2342991 heartbeat_check: no
> > reply from [*:*:*:5::13]:6875 osd.53 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:33.194 7f41a1023700 -1 osd.43 2342991 heartbeat_check: no
> > reply from [*:*:*:5::13]:6894 osd.54 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:33.194 7f41a1023700 -1 osd.43 2342991 heartbeat_check: no
> > reply from [*:*:*:5::14]:6838 osd.71 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:34.034 7f419480a700 0 log_channel(cluster) log [WRN] :
> > Monitor daemon marked osd.43 down, but it is still running
> > 2020-01-15 12:06:34.034 7f419480a700 0 log_channel(cluster) log [DBG] :
> > map e2342992 wrongly marked me down at e2342992
> > 2020-01-15 12:06:34.034 7f419480a700 1 osd.43 2342992
> > start_waiting_for_healthy
> > 2020-01-15 12:06:34.198 7f41a1023700 -1 osd.43 2342992 heartbeat_check: no
> > reply from [*:*:*:5::13]:6854 osd.49 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:34.198 7f41a1023700 -1 osd.43 2342992 heartbeat_check: no
> > reply from [*:*:*:5::13]:6834 osd.51 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:34.198 7f41a1023700 -1 osd.43 2342992 heartbeat_check: no
> > reply from [*:*:*:5::13]:6862 osd.52 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:34.198 7f41a1023700 -1 osd.43 2342992 heartbeat_check: no
> > reply from [*:*:*:5::13]:6875 osd.53 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216 (oldest deadline 2020-01-15 12:06:31.411216)
> > 2020-01-15 12:06:34.198 7f41a1023700 -1 osd.43 2342992 heartbeat_check: no
> > reply from [*:*:*:5::13]:6894 osd.54 ever on either front or back, first
> > ping sent 2020-01-15 12:06:1
> > 1.411216
