--- Begin Message ---
Hello all,
for me the problem unfortunately still exist on my RHEL 7.7 with latest
Netdisco 2.0440013.
Lots of jobs stucked.
Pollers stop after some time, even there are hundreds of queued jobs.
# systemctl status netdisco-backend
● netdisco-backend.service - Netdisco Backend Service
Loaded: loaded (/etc/systemd/system/netdisco-backend.service; enabled;
vendor preset: disabled)
Active: active (running) since Wed 2020-02-05 15:10:36 CET
Main PID: 2921 (netdisco-backen)
CGroup: /system.slice/netdisco-backend.service
├─ 2921 netdisco-backend
├─ 2923 nd2: master
├─ 4289 nd2: #1 sched: idle
└─47188 nd2: #2 mgr: idle
Jan 06 11:10:33 frdv02745.emea.zf-world.com systemd[1]: Starting Netdisco
Backend Service...
Jan 06 11:10:36 frdv02745.emea.zf-world.com netdisco-backend[1802]: config
watcher: watching /opt/netdisco/environments for updates.
Jan 06 11:10:36 frdv02745.emea.zf-world.com netdisco-backend[1802]: Netdisco
Backend [Started]
Jan 06 11:10:36 frdv02745.emea.zf-world.com systemd[1]: Started Netdisco
Backend Service.
A restart solves the situation for some time:
# systemctl restart netdisco-backend
# systemctl status netdisco-backend
● netdisco-backend.service - Netdisco Backend Service
Loaded: loaded (/etc/systemd/system/netdisco-backend.service; enabled;
vendor preset: disabled)
Active: active (running) since Thu 2020-02-06 10:05:26 CET
Process: 29965 ExecStop=/opt/netdisco/bin/netdisco-backend stop (code=exited,
status=0/SUCCESS)
Process: 29975 ExecStart=/opt/netdisco/bin/netdisco-backend start
(code=exited, status=0/SUCCESS)
Main PID: 29977 (netdisco-backen)
CGroup: /system.slice/netdisco-backend.service
├─29977 netdisco-backend
├─29978 nd2: master
├─29986 nd2: #1 sched: idle
├─29987 nd2: #2 mgr: idle
├─29989 nd2: #4 poll: #82795: discover 10.156.1.156
├─29990 nd2: #5 poll: #82615: discover 10.131.112.112
├─29991 nd2: #6 poll: #82720: discover 10.83.174.20
├─29992 nd2: #7 poll: #83183: discover 10.128.41.254
├─29993 nd2: #8 poll: #82634: discover 10.139.6.254
├─29995 nd2: #10 poll: #82828: discover 10.140.24.254
├─29996 nd2: #11 poll: #82665: discover 10.81.171.21
├─29997 nd2: #12 poll: #83181: discover 10.80.180.30
├─29998 nd2: #13 poll: #83128: discover 10.138.77.2
├─29999 nd2: #14 poll: #82749: discover 10.128.11.252
├─30000 nd2: #15 poll: #83105: discover 10.81.67.23
├─30001 nd2: #16 poll: #83247: discover 10.128.4.254
├─30002 nd2: #17 poll: #82754: discover 10.24.4.16
├─30003 nd2: #18 poll: #82685: discover 10.128.74.254
├─30029 nd2: #9 poll: #83580: discover 10.139.81.254
└─30030 nd2: #3 poll: #83635: discover 10.140.24.253
Jan 17 10:05:26 frdv02745.emea.zf-world.com systemd[1]: Starting Netdisco
Backend Service...
Jan 17 10:05:26 frdv02745.emea.zf-world.com netdisco-backend[29975]: Netdisco
Backend [Started]
Jan 17 10:05:26 frdv02745.emea.zf-world.com netdisco-backend[29975]: config
watcher: watching /opt/netdisco/environments for updates.
Jan 17 10:05:26 frdv02745.emea.zf-world.com systemd[1]: Started Netdisco
Backend Service.
As recommended in the install guide I'm using PGtune
(https://pgtune.leopard.in.ua) to tune my PostgreSQL configuration.
I'm wondering what "DB Type" fits best for Netdisco? How do I calculate the
"Number of ( max. database) Connections" required for Netdisco? I'm guessing
the value for "max_connections" must be at least equal better greater than the
number of parallel Netdisco pollers?
Thanks,
Tobias
-----Ursprüngliche Nachricht-----
Von: marco via netdisco-users <[email protected]>
Gesendet: Donnerstag, 6. Februar 2020 09:55
An: Oliver Gorwits <[email protected]>; Ricardo Stella <[email protected]>
Cc: [email protected]
Betreff: Re: [Netdisco] scheduled jobs stop
Hi Oliver
2.0440013 is running
I'll inform you
Thank you
Marco
> Il 4 febbraio 2020 alle 22.36 Oliver Gorwits <[email protected]> ha scritto:
>
> Hi,
> I have released Netdisco 2.044013 which uses a different configuration of the
> MCE job queue handler, on the advice of the MCE developer. This may or may
> not improve things (I've not been able to reproduce the bug), but it would be
> great if feedback comes soon, to let me know if this was a good move!
> many thanksOliver.
> On Tue, 4 Feb 2020 at 15:50, Ricardo Stella < [email protected]> wrote:
> > This is happening on Redhat 7.7 but also on the old instance we were
> > migrating out from which is running Redhat 6.10.
> > On the older instance, we were running version 2.040006 since March 4 2019.
> > I Started the migration to a new instance (DB dump and import on new VM)
> > around January 17th this year.
> > On January 28, I upgraded the old instance to 2.044011 and started to see
> > the same problems as we are experiencing on the new VM. The newer version
> > of MCE:Queue is causing the problems?
> > Hope this helps - Ricardo.
> > On Tue, Feb 4, 2020 at 10:33 AM Oliver Gorwits < [email protected]> wrote:
> > > Can you report back with your operating systems, please?
> > > Many thanks,
> > > On Tue, 4 Feb 2020 at 15:20, Ricardo Stella < [email protected]> wrote:
> > > > Thanks - same issues here. A couple of errors during the last 24 hours
> > > > since I restarted it as the queue was not doing anything over the
> > > > weekend. These are just a few:
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 25006.
> > > > Argument "" isn't numeric in read at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> > > > offset 1 of input at srl_decoder.c line 580 at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 30713.
> > > > Argument "" isn't numeric in read at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> > > > offset 1 of input at srl_decoder.c line 580 at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 48775.
> > > > Argument "" isn't numeric in read at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> > > > offset 1 of input at srl_decoder.c line 580 at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 49491.
> > > >
> > > >
> > > > On Tue, Feb 4, 2020 at 8:12 AM Oliver Gorwits < [email protected]> wrote:
> > > > > Hi Marco
> > > > > I have emailed the developer of the MCE distribution to ask,
> > > > > as I think this is outside of Netdisco's domain, regardsOliver.
> > > > > On Tue, 4 Feb 2020 at 11:36, < [email protected]> wrote:
> > > > > > Hi Oliver
> > > > > >
> > > > > >
> > > > > > It stop again after 4 days
> > > > > >
> > > > > > [14758] 2020-01-30 11:14:30 debug [172.17.121.2] arpnip -
> > > > > > processed 0 IPv6 Neighbor Cache entries [14758] 2020-01-30
> > > > > > 11:14:30 info pol (3): wrapping up arpnip job(22433622) - status
> > > > > > done at Thu Jan 30 12:14:30 2020 Argument "PID_14758" isn't numeric
> > > > > > in abs at /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line
> > > > > > 206, line 128834.
> > > > > > Can't call method "_mce_m_pending" on an undefined value at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 679, line 128835.
> > > > > > [14900] 2020-01-30 13:50:47 warn App::Netdisco 2.044004
> > > > > > backend [14900] 2020-01-30 13:50:47 info resolving backend
> > > > > > hostname...
> > > > > > *************
> > > > > > [14904] 2020-02-03 19:13:41 info mgr (2): job 22463635
> > > > > > booked out for this processing node [14904] 2020-02-03 19:13:41
> > > > > > debug mgr (2): sleeping now...
> > > > > > Argument "_12455" isn't numeric in read at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 204, line
> > > > > > 476776.
> > > > > > Argument
> > > > > > "=rl^D\0A,{App::Netdisco::Backend::Job(*^Ok_statuslist@f..." isn't
> > > > > > numeric in abs at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line
> > > > > > 476776.
> > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > document. at offset 1 of input at srl_decoder.c line 580 at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 480, line 476779.
> > > > > > [12684] 2020-02-04 08:13:38 warn App::Netdisco 2.044004
> > > > > > backend
> > > > > >
> > > > > > It seems to occur randomly, but reading in the log I see
> > > > > > that Argument "SOMETHING" isn't numeric in read at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/SOMETHING.pm
> > > > > >
> > > > > > occur sometimes but usually don't zombies netdisco-backend
> > > > > >
> > > > > > it stop after that sequence [14758] 2020-01-30 11:14:30
> > > > > > info pol (3): wrapping up arpnip job(22433622) - status done
> > > > > > at Thu Jan 30 12:14:30 2020 Argument "PID_14758" isn't numeric in
> > > > > > abs at /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206,
> > > > > > line 128834.
> > > > > >
> > > > > > [12455] 2020-02-03 19:13:41 info pol (3): wrapping up
> > > > > > arpnip job(22463592) - status done at Mon Feb 3 20:13:41 2020 ...
> > > > > > Argument "_12455" isn't numeric in read at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 204, line
> > > > > > 476776.
> > > > > >
> > > > > > just my two cents
> > > > > >
> > > > > > anyway can you suggest me how increase size of log?
> > > > > > cause in debugging mode the 7 files isn't enough for 2 days
> > > > > >
> > > > > > Thank you all
> > > > > > Marco
> > > > > >
> > > > > > > Il 30 gennaio 2020 alle 20.31 Oliver Gorwits < [email protected]>
> > > > > > > ha scritto:
> > > > > > >
> > > > > > > I was looking to see if the issue is related to an upstream
> > > > > > > library change, rather than in Netdisco.
> > > > > > > Mainly because I'm scratching my head trying to work out what
> > > > > > > would cause this, and I can't yet reproduce it.
> > > > > > > On Wed, 29 Jan 2020 at 16:00, Ricardo Stella < [email protected]>
> > > > > > > wrote:
> > > > > > > > Almost there...
> > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv perl -MSereal\ 999
> > > > > > > > -e 1 Sereal version 999 required--this is only version 4.007.
> > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv perl -MMCE::Queue\
> > > > > > > > 999 -e 1 MCE::Queue version 999 required--this is only version
> > > > > > > > 1.865.
> > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv cpanm Sereal MCE
> > > > > > > > Sereal is up to date. (4.007) MCE is up to date. (1.865)
> > > > > > > >
> > > > > > > > I assume we are trying to delete them and force download?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 29, 2020 at 10:52 AM Oliver Gorwits <
> > > > > > > > [email protected]> wrote:
> > > > > > > > > Sorry, my apologies, yes you would need to add "
> > > > > > > > > ~/bin/localenv" to the start of all those commands, I
> > > > > > > > > believe
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, 29 Jan 2020 at 15:17, Ricardo Stella <
> > > > > > > > > [email protected]> wrote:
> > > > > > > > > > Running as the netdisco user, I'm getting:
> > > > > > > > > > Can't locate Sereal.pm in @INC (@INC contains:
> > > > > > > > > > /usr/local/lib64/perl5 /usr/local/share/perl5
> > > > > > > > > > /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl
> > > > > > > > > > /usr/lib64/perl5 /usr/share/perl5 .).
> > > > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > > >
> > > > > > > > > > Does it need --local-lib ~/perl5 or ~/bin/localenv first?
> > > > > > > > > > And --notest?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jan 29, 2020 at 9:47 AM Oliver Gorwits <
> > > > > > > > > > [email protected]> wrote:
> > > > > > > > > > > Hi Ricardo
> > > > > > > > > > > Please can you also run:perl -MSereal\ 999 -e
> > > > > > > > > > > 1perl -MMCE::Queue\ 999 -e 1 Then runcpanm Sereal
> > > > > > > > > > > MCE and then let us know if the problem is still there?
> > > > > > > > > > > thanks,oliver.
> > > > > > > > > > > On Wed, 29 Jan 2020 at 14:15, Ricardo Stella <
> > > > > > > > > > > [email protected]> wrote:
> > > > > > > > > > > > Well, it's definitely a bug with the latest versions.
> > > > > > > > > > > > I upgraded the original instance I had which was
> > > > > > > > > > > > running fine under 2.040006 since March of last year.
> > > > > > > > > > > > This one also is exhibiting the same issues with jobs
> > > > > > > > > > > > queued since 5:30pm yesterday.
> > > > > > > > > > > > Error logs on that instance since last restart
> > > > > > > > > > > > yesterday afternoon are:
> > > > > > > > > > > > [7901] 2020-01-28 16:03:03 warn App::Netdisco
> > > > > > > > > > > > 2.044011 backend Argument "" isn't numeric in read at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439,
> > > > > > > > > > > > <$__ANONIO__> line 1.
> > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > > > > > > > document. at offset 1 of input at srl_decoder.c line
> > > > > > > > > > > > 580 at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > 1445, <$__ANONIO__> line 1.
> > > > > > > > > > > > Argument
> > > > > > > > > > > > "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > > > > > > > > > isn't numeric in int at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484,
> > > > > > > > > > > > <$__ANONIO__> line 1753.
> > > > > > > > > > > > Argument
> > > > > > > > > > > > "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > > > > > > > > > isn't numeric in int at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484,
> > > > > > > > > > > > <$__ANONIO__> line 15984.
> > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439,
> > > > > > > > > > > > <$__ANONIO__> line 1.
> > > > > > > > > > > > Can't call method "status" without a package or object
> > > > > > > > > > > > reference at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/App/Netdisco/Backend/Role/Poller.pm
> > > > > > > > > > > > line 38, <$__ANONIO__> line 1.
> > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439,
> > > > > > > > > > > > <$__ANONIO__> line 1.
> > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > > > > > > > document. at offset 1 of input at srl_decoder.c line
> > > > > > > > > > > > 580 at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > 1445, <$__ANONIO__> line 1.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 28, 2020 at 11:18 AM Ricardo Stella <
> > > > > > > > > > > > [email protected]> wrote:
> > > > > > > > > > > > > And just noticed that there's a newer version out
> > > > > > > > > > > > > there. Updated the new instance (including wiping the
> > > > > > > > > > > > > perl5 directory) and right after I started it, I got
> > > > > > > > > > > > > an error message. The old one was also updated but
> > > > > > > > > > > > > it's not giving me any errors so far.
> > > > > > > > > > > > > [8849] 2020-01-28 16:13:41 warn App::Netdisco
> > > > > > > > > > > > > 2.044011 backend Argument "" isn't numeric in read at
> > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > 1439, line 1.
> > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > > > > > > > > document. at offset 1 of input at srl_decoder.c line
> > > > > > > > > > > > > 580 at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > line 1445, line 1.
> > > > > > > > > > > > > Argument
> > > > > > > > > > > > > "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > > > > > > > > > > isn't numeric in int at
> > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > 1484, line 32.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 28, 2020 at 9:56 AM Ricardo Stella <
> > > > > > > > > > > > > [email protected]> wrote:
> > > > > > > > > > > > > > Same here...
> > > > > > > > > > > > > > backend status thinks it's running but jobs are
> > > > > > > > > > > > > > queued since last night and not running. Here are
> > > > > > > > > > > > > > the errors since last restart yesterday:
> > > > > > > > > > > > > > [24657] 2020-01-27 16:00:58 warn
> > > > > > > > > > > > > > App::Netdisco 2.044009 backend Argument "" isn't
> > > > > > > > > > > > > > numeric in read at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > 1439, line 1.
> > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid
> > > > > > > > > > > > > > Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > 1445, line 1.
> > > > > > > > > > > > > > Argument "=M-srl^D\0A,n yesterday after few hour.
> > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > [5754] 2020-01-27 17:06:59 debug -> run worker
> > > > > > > > > > > > > > main/wirelessnodes/100
> > > > > > > > > > > > > > [5754] 2020-01-27 17:06:59 info pol (3):
> > > > > > > > > > > > > > wrapping up macsuck job(22425208) - status done at
> > > > > > > > > > > > > > Mon Jan 27 18:06:59 2020
> > > > > > > > > > > > > > [5750] 2020-01-27 17:06:59 debug
> > > > > > > > > > > > > > [172.17.119.6] macsuck - port 1:43 vlan unknown : 1
> > > > > > > > > > > > > > nodes
> > > > > > > > > > > > > > Argument "PID_5754" isn't numeric in abs at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm
> > > > > > > > > > > > > > line 206, line 32948.
> > > > > > > > > > > > > > Can't call method "_mce_m_pending" on an
> > > > > > > > > > > > > > undefined value at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > 679, line 32949.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I activated debug, it seems that some scheduled
> > > > > > > > > > > > > > jobs (macsuck, discoverall etc.) cause the error
> > > > > > > > > > > > > > "Argument "PID_####" isn't numeric " and it zombies
> > > > > > > > > > > > > > netdisco-backend child
> > > > > > > > > > > > > > ps aux | grep netd
> > > > > > > > > > > > > > netdisco 3428 0.0 0.3 22840 15848 ?
> > > > > > > > > > > > > > S gen27 2:05 netdisco-backend
> > > > > > > > > > > > > > netdisco 3429 0.0 0.0 0 0 ?
> > > > > > > > > > > > > > Z gen27 0:15 [nd2: master]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I can't say if it is caused by my new
> > > > > > > > > > > > > > setup/configuration or something else
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Marco
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Il 27 gennaio 2020 alle 17.03 Ricardo Stella <
> > > > > > > > > > > > > > > [email protected]> ha scritto:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Also happening here. I also had exported the DB
> > > > > > > > > > > > > > > in order to install on a new VM with new OS. Had
> > > > > > > > > > > > > > > a couple of problems that I posted but had this
> > > > > > > > > > > > > > > same error on the logs.
> > > > > > > > > > > > > > > Noticed all jobs queued for a couple of days and
> > > > > > > > > > > > > > > nothing running.
> > > > > > > > > > > > > > > Last message on logs was:
> > > > > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > > 1439, line 1.
> > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid
> > > > > > > > > > > > > > > Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > > 1445, line 1.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Restarting it seems to get the jobs running
> > > > > > > > > > > > > > > again.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Jan 27, 2020 at 10:54 AM marco via
> > > > > > > > > > > > > > > netdisco-users <
> > > > > > > > > > > > > > > [email protected]> wrote:
> > > > > > > > > > > > > > > > Hi there
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I had set up a new ND2 host on debian
> > > > > > > > > > > > > > > > buster some weeks ago for experimental
> > > > > > > > > > > > > > > > purpose I have another ND2 host up and
> > > > > > > > > > > > > > > > running since years
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Software Version
> > > > > > > > > > > > > > > > App::Netdisco 2.44.4
> > > > > > > > > > > > > > > > SNMP::Info 3.70
> > > > > > > > > > > > > > > > DB Schema 61
> > > > > > > > > > > > > > > > PostgreSQL 12.00.1
> > > > > > > > > > > > > > > > Perl 5.28.1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I restore db from another ND2 and copy
> > > > > > > > > > > > > > > > deployment.yml It worked
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > But I noticed that it stops running the
> > > > > > > > > > > > > > > > scheduled jobs after some times (days) I
> > > > > > > > > > > > > > > > had to restart netdisco-backend,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > here some info I collect
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > from netdisco-backend.log
> > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 debug mgr (2):
> > > > > > > > > > > > > > > > getting potential jobs for 1 workers
> > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 debug
> > > > > > > > > > > > > > > > [172.17.185.50] arpnip - processed 373 ARP
> > > > > > > > > > > > > > > > Cache entries
> > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 debug
> > > > > > > > > > > > > > > > [172.17.185.50] arpnip - processed 0 IPv6
> > > > > > > > > > > > > > > > Neighbor Cache entries
> > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 info pol (3):
> > > > > > > > > > > > > > > > wrapping up arpnip job(22423168) - status done
> > > > > > > > > > > > > > > > at Fri Jan 24 16:15:18 2020
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 debug getsome:
> > > > > > > > > > > > > > > > cancelled 0E0 duplicate(s) of job 22423235
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 info mgr (2):
> > > > > > > > > > > > > > > > job 22423235 booked out for this processing
> > > > > > > > > > > > > > > > node
> > > > > > > > > > > > > > > > Argument "PID_2700" isn't numeric in read
> > > > > > > > > > > > > > > > at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 477, line 31470.
> > > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a
> > > > > > > > > > > > > > > > valid Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 480, line 31470.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > root@deb-netdisco:~# systemctl status
> > > > > > > > > > > > > > > > netdisco-backend.service
> > > > > > > > > > > > > > > > ● netdisco-backend.service - Netdisco
> > > > > > > > > > > > > > > > Backend Service
> > > > > > > > > > > > > > > > Loaded: loaded
> > > > > > > > > > > > > > > > (/etc/systemd/system/netdisco-backend.service;
> > > > > > > > > > > > > > > > enabled; vendor preset: enabled)
> > > > > > > > > > > > > > > > Active: active (running) since Fri
> > > > > > > > > > > > > > > > 2020-01-24 09:53:03 CET; 3 days ago
> > > > > > > > > > > > > > > > Process: 110
> > > > > > > > > > > > > > > > ExecStart=/home/netdisco/bin/netdisco-backend
> > > > > > > > > > > > > > > > start (code=exited, status=0/SUCCESS)
> > > > > > > > > > > > > > > > Main PID: 216 (netdisco-backen)
> > > > > > > > > > > > > > > > Tasks: 2 (limit: 4915)
> > > > > > > > > > > > > > > > Memory: 143.0M
> > > > > > > > > > > > > > > > CGroup:
> > > > > > > > > > > > > > > > /system.slice/netdisco-backend.service
> > > > > > > > > > > > > > > > └─216 netdisco-backend
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > gen 24 09:53:02 deb-netdisco systemd[1]:
> > > > > > > > > > > > > > > > Starting Netdisco Backend Service...
> > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: Netdisco Backend
> > > > > > > > > > > > > > > > [Started]
> > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: config watcher: watching
> > > > > > > > > > > > > > > > /home/netdisco/environments for updates.
> > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco systemd[1]:
> > > > > > > > > > > > > > > > Started Netdisco Backend Service.
> > > > > > > > > > > > > > > > gen 24 10:01:48 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: --
> > > > > > > > > > > > > > > > /home/netdisco/environments/deployment.yml
> > > > > > > > > > > > > > > > updated.
> > > > > > > > > > > > > > > > gen 24 10:01:48 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: config watcher: sending
> > > > > > > > > > > > > > > > TERM to the server (pid:217)...
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > root@deb-netdisco:~# ps aux | grep netd
> > > > > > > > > > > > > > > > netdisco 216 0.0 0.3 22840 16008 ?
> > > > > > > > > > > > > > > > S gen24 6:19 netdisco-backend
> > > > > > > > > > > > > > > > netdisco 281 0.0 0.3 20744 13680 ?
> > > > > > > > > > > > > > > > S gen24 0:00 perl
> > > > > > > > > > > > > > > > /home/netdisco/bin/netdisco-web start
> > > > > > > > > > > > > > > > netdisco 282 0.0 0.3 22152 16696 ?
> > > > > > > > > > > > > > > > S gen24 0:47 starman master
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 372 0.0 0.0 0 0 ?
> > > > > > > > > > > > > > > > Z gen24 0:16 [nd2: master]
> > > > > > > > > > > > > > > > netdisco 373 0.0 2.7 135148 117200 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 374 0.0 2.8 136000 118000 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 375 0.0 2.7 133744 115940 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 376 0.0 2.8 137420 119504 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 377 0.0 2.7 133792 115996 ?
> > > > > > > > > > > > > > > > S gen24 0:05 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > root 3405 0.0 0.0 6096 824 pts/0
> > > > > > > > > > > > > > > > S+ 10:59 0:00 grep netd
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > after stop and start
> > > > > > > > > > > > > > > > root@deb-netdisco:~# systemctl start
> > > > > > > > > > > > > > > > netdisco-backend.service
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > it seems to work again
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 info mgr (2):
> > > > > > > > > > > > > > > > job 22423235 booked out for this processing
> > > > > > > > > > > > > > > > node
> > > > > > > > > > > > > > > > Argument "PID_2700" isn't numeric in read
> > > > > > > > > > > > > > > > at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 477, line 31470.
> > > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a
> > > > > > > > > > > > > > > > valid Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 480, line 31470.
> > > > > > > > > > > > > > > > [3429] 2020-01-27 10:10:08 warn
> > > > > > > > > > > > > > > > App::Netdisco 2.044004 backend
> > > > > > > > > > > > > > > > [3429] 2020-01-27 10:10:08 info resolving
> > > > > > > > > > > > > > > > backend hostname...
> > > > > > > > > > > > > > > > [3433] 2020-01-27 10:10:08 info applying
> > > > > > > > > > > > > > > > role Scheduler to worker 1
> > > > > > > > > > > > > > > > [3436] 2020-01-27 10:10:08 info applying
> > > > > > > > > > > > > > > > role Poller to worker 4
> > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ________________________________________
> > > > > > > > > > > > > > > > _______ Netdisco mailing list
> > > > > > > > > > > > > > > > [email protected]
> > > > > > > > > > > > > > > > https://sourceforge.net/p/netdisco/mailm
> > > > > > > > > > > > > > > > an/netdisco-users/
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > > > ==========================================
> > > > > > > > > > > > > > > ======
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > > ============================================
> > > > > > > > > > > > > > ====
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > ==============================================
> > > > > > > > > > > > > ==
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > ================================================
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > Netdisco mailing list
> > > > > > > > > > > > [email protected]
> > > > > > > > > > > > https://sourceforge.net/p/netdisco/mailman/netdi
> > > > > > > > > > > > sco-users/
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > ================================================
> > > > > > > >
> > > > > > > > --
> > > > > > > > °((( = (( ===°°° (((
> > > > > > > > ================================================
> > > >
> > > > --
> > > > °((( = (( ===°°° (((
> > > > ================================================
> >
> > --
> > °((( = (( ===°°° (((
> > ================================================
_______________________________________________
Netdisco mailing list
[email protected]
https://sourceforge.net/p/netdisco/mailman/netdisco-users/
--- End Message ---
