--- Begin Message ---
hm, how did you enter those jobs?
if you did via netdisco-do --enqueue the jobs will be queued until the next
scheduled run of that action type.
about the postgres thing:
netdisco will make 1 db connection per worker (only when they are running), 1
for the scheduler, i think 1 for each web worker, and perhaps 1 for the master
& manager. 100 connections should be more than sufficient for just netdisco, if
you need more you might also need a decent dba :)
we updated the manpage a few months ago with pgtune alternatives since that
project seems to be dead. i prefer https://pgtune.leopard.in.ua/#/ . as type
"web application" should be fine. keep in mind these just give an educated
guess for settings.
dunno what your target is for netdisco, but i'm running a test and production
setup on the same db with around 1200 devices, 21000 interfaces and 20000
nodes. netdisco is set to 20 workers (test and prod combined), schedules are
spread so different actions don't run at the same time:
schedule:
discoverall:
when: '5 3,7,11,15,19,23 * * *'
macwalk:
when:
min: 55
hour: '*/4'
arpwalk:
when:
min: 5
hour: '*/4'
nbtwalk: null
expire:
when: '30 23 * * *'
on opensuse leap 15.1 these are the only settings i changed:
max_connections = 100
shared_buffers = 1GB
maintenance_work_mem = 256MB
effective_io_concurrency = 10
# this one is for debugging, shouldn't be needed in prod setup
track_activity_query_size = 8192
// nick
> -----Original Message-----
> From: Gerlach Tobias DLN FIII31 [mailto:[email protected]]
> Sent: Thursday, February 6, 2020 10:29
> To: [email protected]
> Cc: Oliver Gorwits <[email protected]>
> Subject: Re: [Netdisco] scheduled jobs stop
>
> Hello all,
>
> for me the problem unfortunately still exist on my RHEL 7.7 with latest
> Netdisco 2.0440013.
> Lots of jobs stucked.
> Pollers stop after some time, even there are hundreds of queued jobs.
>
> # systemctl status netdisco-backend
> ● netdisco-backend.service - Netdisco Backend Service
> Loaded: loaded (/etc/systemd/system/netdisco-backend.service; enabled;
> vendor preset: disabled)
> Active: active (running) since Wed 2020-02-05 15:10:36 CET
> Main PID: 2921 (netdisco-backen)
> CGroup: /system.slice/netdisco-backend.service
> ├─ 2921 netdisco-backend
> ├─ 2923 nd2: master
> ├─ 4289 nd2: #1 sched: idle
> └─47188 nd2: #2 mgr: idle
>
> Jan 06 11:10:33 frdv02745.emea.zf-world.com systemd[1]: Starting Netdisco
> Backend Service...
> Jan 06 11:10:36 frdv02745.emea.zf-world.com netdisco-backend[1802]: config
> watcher: watching /opt/netdisco/environments for updates.
> Jan 06 11:10:36 frdv02745.emea.zf-world.com netdisco-backend[1802]:
> Netdisco Backend [Started]
> Jan 06 11:10:36 frdv02745.emea.zf-world.com systemd[1]: Started Netdisco
> Backend Service.
>
> A restart solves the situation for some time:
>
> # systemctl restart netdisco-backend
>
> # systemctl status netdisco-backend
> ● netdisco-backend.service - Netdisco Backend Service
> Loaded: loaded (/etc/systemd/system/netdisco-backend.service; enabled;
> vendor preset: disabled)
> Active: active (running) since Thu 2020-02-06 10:05:26 CET
> Process: 29965 ExecStop=/opt/netdisco/bin/netdisco-backend stop
> (code=exited, status=0/SUCCESS)
> Process: 29975 ExecStart=/opt/netdisco/bin/netdisco-backend start
> (code=exited, status=0/SUCCESS)
> Main PID: 29977 (netdisco-backen)
> CGroup: /system.slice/netdisco-backend.service
> ├─29977 netdisco-backend
> ├─29978 nd2: master
> ├─29986 nd2: #1 sched: idle
> ├─29987 nd2: #2 mgr: idle
> ├─29989 nd2: #4 poll: #82795: discover 10.156.1.156
> ├─29990 nd2: #5 poll: #82615: discover 10.131.112.112
> ├─29991 nd2: #6 poll: #82720: discover 10.83.174.20
> ├─29992 nd2: #7 poll: #83183: discover 10.128.41.254
> ├─29993 nd2: #8 poll: #82634: discover 10.139.6.254
> ├─29995 nd2: #10 poll: #82828: discover 10.140.24.254
> ├─29996 nd2: #11 poll: #82665: discover 10.81.171.21
> ├─29997 nd2: #12 poll: #83181: discover 10.80.180.30
> ├─29998 nd2: #13 poll: #83128: discover 10.138.77.2
> ├─29999 nd2: #14 poll: #82749: discover 10.128.11.252
> ├─30000 nd2: #15 poll: #83105: discover 10.81.67.23
> ├─30001 nd2: #16 poll: #83247: discover 10.128.4.254
> ├─30002 nd2: #17 poll: #82754: discover 10.24.4.16
> ├─30003 nd2: #18 poll: #82685: discover 10.128.74.254
> ├─30029 nd2: #9 poll: #83580: discover 10.139.81.254
> └─30030 nd2: #3 poll: #83635: discover 10.140.24.253
>
> Jan 17 10:05:26 frdv02745.emea.zf-world.com systemd[1]: Starting Netdisco
> Backend Service...
> Jan 17 10:05:26 frdv02745.emea.zf-world.com netdisco-backend[29975]:
> Netdisco Backend [Started]
> Jan 17 10:05:26 frdv02745.emea.zf-world.com netdisco-backend[29975]:
> config watcher: watching /opt/netdisco/environments for updates.
> Jan 17 10:05:26 frdv02745.emea.zf-world.com systemd[1]: Started Netdisco
> Backend Service.
>
>
> As recommended in the install guide I'm using PGtune
> (https://pgtune.leopard.in.ua) to tune my PostgreSQL configuration.
>
> I'm wondering what "DB Type" fits best for Netdisco? How do I calculate the
> "Number of ( max. database) Connections" required for Netdisco? I'm
> guessing the value for "max_connections" must be at least equal better
> greater than the number of parallel Netdisco pollers?
>
> Thanks,
> Tobias
>
>
> -----Ursprüngliche Nachricht-----
> Von: marco via netdisco-users <[email protected]>
> Gesendet: Donnerstag, 6. Februar 2020 09:55
> An: Oliver Gorwits <[email protected]>; Ricardo Stella <[email protected]>
> Cc: [email protected]
> Betreff: Re: [Netdisco] scheduled jobs stop
>
> Hi Oliver
>
> 2.0440013 is running
>
> I'll inform you
>
> Thank you
>
> Marco
>
> > Il 4 febbraio 2020 alle 22.36 Oliver Gorwits <[email protected]> ha scritto:
> >
> > Hi,
> > I have released Netdisco 2.044013 which uses a different configuration of
> the MCE job queue handler, on the advice of the MCE developer. This may or
> may not improve things (I've not been able to reproduce the bug), but it would
> be great if feedback comes soon, to let me know if this was a good move!
> > many thanksOliver.
> > On Tue, 4 Feb 2020 at 15:50, Ricardo Stella < [email protected]> wrote:
> > > This is happening on Redhat 7.7 but also on the old instance we were
> migrating out from which is running Redhat 6.10.
> > > On the older instance, we were running version 2.040006 since March 4
> 2019. I Started the migration to a new instance (DB dump and import on new
> VM) around January 17th this year.
> > > On January 28, I upgraded the old instance to 2.044011 and started to see
> the same problems as we are experiencing on the new VM. The newer version
> of MCE:Queue is causing the problems?
> > > Hope this helps - Ricardo.
> > > On Tue, Feb 4, 2020 at 10:33 AM Oliver Gorwits < [email protected]>
> wrote:
> > > > Can you report back with your operating systems, please?
> > > > Many thanks,
> > > > On Tue, 4 Feb 2020 at 15:20, Ricardo Stella < [email protected]> wrote:
> > > > > Thanks - same issues here. A couple of errors during the last 24 hours
> since I restarted it as the queue was not doing anything over the weekend.
> These are just a few:
> > > > > Argument "=M-
> srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..." isn't numeric in int
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484, line 25006.
> > > > > Argument "" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > > Argument "=M-
> srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..." isn't numeric in int
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484, line 30713.
> > > > > Argument "" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > > Argument "=M-
> srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..." isn't numeric in int
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484, line 48775.
> > > > > Argument "" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > > Argument "=M-
> srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..." isn't numeric in int
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484, line 49491.
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2020 at 8:12 AM Oliver Gorwits < [email protected]>
> wrote:
> > > > > > Hi Marco
> > > > > > I have emailed the developer of the MCE distribution to ask,
> > > > > > as I think this is outside of Netdisco's domain, regardsOliver.
> > > > > > On Tue, 4 Feb 2020 at 11:36, < [email protected]> wrote:
> > > > > > > Hi Oliver
> > > > > > >
> > > > > > >
> > > > > > > It stop again after 4 days
> > > > > > >
> > > > > > > [14758] 2020-01-30 11:14:30 debug [172.17.121.2] arpnip -
> > > > > > > processed 0 IPv6 Neighbor Cache entries [14758] 2020-01-30
> > > > > > > 11:14:30 info pol (3): wrapping up arpnip job(22433622) - status
> done at Thu Jan 30 12:14:30 2020 Argument "PID_14758" isn't numeric in abs
> at /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line 128834.
> > > > > > > Can't call method "_mce_m_pending" on an undefined value at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 679, line 128835.
> > > > > > > [14900] 2020-01-30 13:50:47 warn App::Netdisco 2.044004
> > > > > > > backend [14900] 2020-01-30 13:50:47 info resolving backend
> hostname...
> > > > > > > *************
> > > > > > > [14904] 2020-02-03 19:13:41 info mgr (2): job 22463635
> > > > > > > booked out for this processing node [14904] 2020-02-03 19:13:41
> debug mgr (2): sleeping now...
> > > > > > > Argument "_12455" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 204, line 476776.
> > > > > > > Argument
> "=rl^D\0A,{App::Netdisco::Backend::Job(*^Ok_statuslist@f..." isn't numeric in
> abs at /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line
> 476776.
> > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > > document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 480, line 476779.
> > > > > > > [12684] 2020-02-04 08:13:38 warn App::Netdisco 2.044004
> > > > > > > backend
> > > > > > >
> > > > > > > It seems to occur randomly, but reading in the log I see
> > > > > > > that Argument "SOMETHING" isn't numeric in read at
> > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/SOMETHING.pm
> > > > > > >
> > > > > > > occur sometimes but usually don't zombies netdisco-backend
> > > > > > >
> > > > > > > it stop after that sequence [14758] 2020-01-30 11:14:30
> > > > > > > info pol (3): wrapping up arpnip job(22433622) - status done
> > > > > > > at Thu Jan 30 12:14:30 2020 Argument "PID_14758" isn't numeric
> in abs at /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line
> 128834.
> > > > > > >
> > > > > > > [12455] 2020-02-03 19:13:41 info pol (3): wrapping up
> > > > > > > arpnip job(22463592) - status done at Mon Feb 3 20:13:41 2020 ...
> > > > > > > Argument "_12455" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 204, line 476776.
> > > > > > >
> > > > > > > just my two cents
> > > > > > >
> > > > > > > anyway can you suggest me how increase size of log?
> > > > > > > cause in debugging mode the 7 files isn't enough for 2 days
> > > > > > >
> > > > > > > Thank you all
> > > > > > > Marco
> > > > > > >
> > > > > > > > Il 30 gennaio 2020 alle 20.31 Oliver Gorwits < [email protected]>
> ha scritto:
> > > > > > > >
> > > > > > > > I was looking to see if the issue is related to an upstream
> > > > > > > > library
> change, rather than in Netdisco.
> > > > > > > > Mainly because I'm scratching my head trying to work out what
> would cause this, and I can't yet reproduce it.
> > > > > > > > On Wed, 29 Jan 2020 at 16:00, Ricardo Stella <
> [email protected]> wrote:
> > > > > > > > > Almost there...
> > > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv perl -MSereal\ 999
> > > > > > > > > -e 1 Sereal version 999 required--this is only version 4.007.
> > > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv perl -MMCE::Queue\
> > > > > > > > > 999 -e 1 MCE::Queue version 999 required--this is only version
> 1.865.
> > > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv cpanm Sereal MCE
> > > > > > > > > Sereal is up to date. (4.007) MCE is up to date. (1.865)
> > > > > > > > >
> > > > > > > > > I assume we are trying to delete them and force download?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Jan 29, 2020 at 10:52 AM Oliver Gorwits <
> [email protected]> wrote:
> > > > > > > > > > Sorry, my apologies, yes you would need to add "
> > > > > > > > > > ~/bin/localenv" to the start of all those commands, I
> > > > > > > > > > believe
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, 29 Jan 2020 at 15:17, Ricardo Stella <
> [email protected]> wrote:
> > > > > > > > > > > Running as the netdisco user, I'm getting:
> > > > > > > > > > > Can't locate Sereal.pm in @INC (@INC contains:
> /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl
> /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).
> > > > > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > > > >
> > > > > > > > > > > Does it need --local-lib ~/perl5 or ~/bin/localenv first?
> > > > > > > > > > > And
> --notest?
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jan 29, 2020 at 9:47 AM Oliver Gorwits <
> [email protected]> wrote:
> > > > > > > > > > > > Hi Ricardo
> > > > > > > > > > > > Please can you also run:perl -MSereal\ 999 -e
> > > > > > > > > > > > 1perl -MMCE::Queue\ 999 -e 1 Then runcpanm Sereal
> > > > > > > > > > > > MCE and then let us know if the problem is still there?
> > > > > > > > > > > > thanks,oliver.
> > > > > > > > > > > > On Wed, 29 Jan 2020 at 14:15, Ricardo Stella <
> [email protected]> wrote:
> > > > > > > > > > > > > Well, it's definitely a bug with the latest versions.
> > > > > > > > > > > > > I
> upgraded the original instance I had which was running fine under 2.040006
> since March of last year. This one also is exhibiting the same issues with
> jobs
> queued since 5:30pm yesterday.
> > > > > > > > > > > > > Error logs on that instance since last restart
> > > > > > > > > > > > > yesterday
> afternoon are:
> > > > > > > > > > > > > [7901] 2020-01-28 16:03:03 warn App::Netdisco
> > > > > > > > > > > > > 2.044011 backend Argument "" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, <$__ANONIO__> line
> 1.
> > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, <$__ANONIO__> line
> 1.
> > > > > > > > > > > > > Argument "=M-
> srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..." isn't numeric in int
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484, <$__ANONIO__>
> line 1753.
> > > > > > > > > > > > > Argument "=M-
> srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..." isn't numeric in int
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484, <$__ANONIO__>
> line 15984.
> > > > > > > > > > > > > Argument "" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, <$__ANONIO__> line
> 1.
> > > > > > > > > > > > > Can't call method "status" without a package or object
> reference at
> /home/netdisco/perl5/lib/perl5/App/Netdisco/Backend/Role/Poller.pm line 38,
> <$__ANONIO__> line 1.
> > > > > > > > > > > > > Argument "" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, <$__ANONIO__> line
> 1.
> > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, <$__ANONIO__> line
> 1.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 28, 2020 at 11:18 AM Ricardo Stella <
> [email protected]> wrote:
> > > > > > > > > > > > > > And just noticed that there's a newer version out
> there. Updated the new instance (including wiping the perl5 directory) and
> right
> after I started it, I got an error message. The old one was also updated but
> it's
> not giving me any errors so far.
> > > > > > > > > > > > > > [8849] 2020-01-28 16:13:41 warn App::Netdisco
> > > > > > > > > > > > > > 2.044011 backend Argument "" isn't numeric in read
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > > > > > > > > > > > Argument "=M-
> srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..." isn't numeric in int
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484, line 32.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jan 28, 2020 at 9:56 AM Ricardo Stella <
> [email protected]> wrote:
> > > > > > > > > > > > > > > Same here...
> > > > > > > > > > > > > > > backend status thinks it's running but jobs are
> queued since last night and not running. Here are the errors since last
> restart
> yesterday:
> > > > > > > > > > > > > > > [24657] 2020-01-27 16:00:58 warn
> > > > > > > > > > > > > > > App::Netdisco 2.044009 backend Argument ""
> isn't numeric in read at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> 1439, line 1.
> > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid
> Sereal document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > > > > > > > > > > > > Argument "=M-srl^D\0A,n yesterday after few
> hour.
> > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > [5754] 2020-01-27 17:06:59 debug -> run worker
> main/wirelessnodes/100
> > > > > > > > > > > > > > > [5754] 2020-01-27 17:06:59 info pol (3):
> wrapping up macsuck job(22425208) - status done at Mon Jan 27 18:06:59
> 2020
> > > > > > > > > > > > > > > [5750] 2020-01-27 17:06:59 debug
> [172.17.119.6] macsuck - port 1:43 vlan unknown : 1 nodes
> > > > > > > > > > > > > > > Argument "PID_5754" isn't numeric in abs at
> /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line 32948.
> > > > > > > > > > > > > > > Can't call method "_mce_m_pending" on an
> undefined value at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 679,
> line 32949.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I activated debug, it seems that some scheduled
> jobs (macsuck, discoverall etc.) cause the error "Argument "PID_####" isn't
> numeric " and it zombies netdisco-backend child
> > > > > > > > > > > > > > > ps aux | grep netd
> > > > > > > > > > > > > > > netdisco 3428 0.0 0.3 22840 15848 ?
> > > > > > > > > > > > > > > S
> gen27 2:05 netdisco-backend
> > > > > > > > > > > > > > > netdisco 3429 0.0 0.0 0 0 ?
> > > > > > > > > > > > > > > Z
> gen27 0:15 [nd2: master]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I can't say if it is caused by my new
> > > > > > > > > > > > > > > setup/configuration or something else
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Marco
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Il 27 gennaio 2020 alle 17.03 Ricardo Stella <
> [email protected]> ha scritto:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Also happening here. I also had exported the
> DB in order to install on a new VM with new OS. Had a couple of problems
> that I posted but had this same error on the logs.
> > > > > > > > > > > > > > > > Noticed all jobs queued for a couple of days
> and nothing running.
> > > > > > > > > > > > > > > > Last message on logs was:
> > > > > > > > > > > > > > > > Argument "" isn't numeric in read at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid
> Sereal document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Restarting it seems to get the jobs running
> again.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Jan 27, 2020 at 10:54 AM marco via
> netdisco-users < [email protected]> wrote:
> > > > > > > > > > > > > > > > > Hi there
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I had set up a new ND2 host on debian
> > > > > > > > > > > > > > > > > buster some weeks ago for experimental
> > > > > > > > > > > > > > > > > purpose I have another ND2 host up and
> > > > > > > > > > > > > > > > > running since years
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Software Version
> > > > > > > > > > > > > > > > > App::Netdisco 2.44.4
> > > > > > > > > > > > > > > > > SNMP::Info 3.70
> > > > > > > > > > > > > > > > > DB Schema 61
> > > > > > > > > > > > > > > > > PostgreSQL 12.00.1
> > > > > > > > > > > > > > > > > Perl 5.28.1
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I restore db from another ND2 and copy
> > > > > > > > > > > > > > > > > deployment.yml It worked
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > But I noticed that it stops running the
> > > > > > > > > > > > > > > > > scheduled jobs after some times (days) I
> > > > > > > > > > > > > > > > > had to restart netdisco-backend,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > here some info I collect
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > from netdisco-backend.log
> > > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 debug mgr (2):
> getting potential jobs for 1 workers
> > > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 debug
> [172.17.185.50] arpnip - processed 373 ARP Cache entries
> > > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 debug
> [172.17.185.50] arpnip - processed 0 IPv6 Neighbor Cache entries
> > > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 info pol (3):
> wrapping up arpnip job(22423168) - status done at Fri Jan 24 16:15:18 2020
> > > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 debug getsome:
> cancelled 0E0 duplicate(s) of job 22423235
> > > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 info mgr (2):
> > > > > > > > > > > > > > > > > job
> 22423235 booked out for this processing node
> > > > > > > > > > > > > > > > > Argument "PID_2700" isn't numeric in read
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 477, line 31470.
> > > > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a
> valid Sereal document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 480, line 31470.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > root@deb-netdisco:~# systemctl status
> netdisco-backend.service
> > > > > > > > > > > > > > > > > ● netdisco-backend.service - Netdisco
> Backend Service
> > > > > > > > > > > > > > > > > Loaded: loaded
> (/etc/systemd/system/netdisco-backend.service; enabled; vendor preset:
> enabled)
> > > > > > > > > > > > > > > > > Active: active (running) since Fri
> > > > > > > > > > > > > > > > > 2020-01-24
> 09:53:03 CET; 3 days ago
> > > > > > > > > > > > > > > > > Process: 110
> ExecStart=/home/netdisco/bin/netdisco-backend start (code=exited,
> status=0/SUCCESS)
> > > > > > > > > > > > > > > > > Main PID: 216 (netdisco-backen)
> > > > > > > > > > > > > > > > > Tasks: 2 (limit: 4915)
> > > > > > > > > > > > > > > > > Memory: 143.0M
> > > > > > > > > > > > > > > > > CGroup: /system.slice/netdisco-
> backend.service
> > > > > > > > > > > > > > > > > └─216 netdisco-backend
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > gen 24 09:53:02 deb-netdisco systemd[1]:
> Starting Netdisco Backend Service...
> > > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco netdisco-
> backend[110]: Netdisco Backend
> [Started]
> > > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco netdisco-
> backend[110]: config watcher: watching /home/netdisco/environments for
> updates.
> > > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco systemd[1]:
> Started Netdisco Backend Service.
> > > > > > > > > > > > > > > > > gen 24 10:01:48 deb-netdisco netdisco-
> backend[110]: -- /home/netdisco/environments/deployment.yml updated.
> > > > > > > > > > > > > > > > > gen 24 10:01:48 deb-netdisco netdisco-
> backend[110]: config watcher: sending TERM to the server (pid:217)...
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > root@deb-netdisco:~# ps aux | grep netd
> > > > > > > > > > > > > > > > > netdisco 216 0.0 0.3 22840 16008 ?
> S gen24 6:19 netdisco-backend
> > > > > > > > > > > > > > > > > netdisco 281 0.0 0.3 20744 13680 ?
> S gen24 0:00 perl /home/netdisco/bin/netdisco-web start
> > > > > > > > > > > > > > > > > netdisco 282 0.0 0.3 22152 16696 ?
> S gen24 0:47 starman master --disable-keepalive --user 1001 --group 1001
> /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > > netdisco 372 0.0 0.0 0 0 ?
> > > > > > > > > > > > > > > > > Z
> gen24 0:16 [nd2: master]
> > > > > > > > > > > > > > > > > netdisco 373 0.0 2.7 135148 117200 ?
> S gen24 0:06 starman worker --disable-keepalive --user 1001 --group 1001
> /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > > netdisco 374 0.0 2.8 136000 118000 ?
> S gen24 0:06 starman worker --disable-keepalive --user 1001 --group 1001
> /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > > netdisco 375 0.0 2.7 133744 115940 ?
> S gen24 0:06 starman worker --disable-keepalive --user 1001 --group 1001
> /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > > netdisco 376 0.0 2.8 137420 119504 ?
> S gen24 0:06 starman worker --disable-keepalive --user 1001 --group 1001
> /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > > netdisco 377 0.0 2.7 133792 115996 ?
> S gen24 0:05 starman worker --disable-keepalive --user 1001 --group 1001
> /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > > root 3405 0.0 0.0 6096 824
> > > > > > > > > > > > > > > > > pts/0
> S+ 10:59 0:00 grep netd
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > after stop and start
> > > > > > > > > > > > > > > > > root@deb-netdisco:~# systemctl start
> > > > > > > > > > > > > > > > > netdisco-backend.service
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > it seems to work again
> > > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 info mgr (2):
> > > > > > > > > > > > > > > > > job
> 22423235 booked out for this processing node
> > > > > > > > > > > > > > > > > Argument "PID_2700" isn't numeric in read
> at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 477, line 31470.
> > > > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a
> valid Sereal document. at offset 1 of input at srl_decoder.c line 580 at
> /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 480, line 31470.
> > > > > > > > > > > > > > > > > [3429] 2020-01-27 10:10:08 warn
> App::Netdisco 2.044004 backend
> > > > > > > > > > > > > > > > > [3429] 2020-01-27 10:10:08 info resolving
> backend hostname...
> > > > > > > > > > > > > > > > > [3433] 2020-01-27 10:10:08 info applying
> role Scheduler to worker 1
> > > > > > > > > > > > > > > > > [3436] 2020-01-27 10:10:08 info applying
> role Poller to worker 4
> > > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> ________________________________________
> > > > > > > > > > > > > > > > > _______ Netdisco mailing list
> > > > > > > > > > > > > > > > > [email protected]
> > > > > > > > > > > > > > > > > https://sourceforge.net/p/netdisco/mailm
> > > > > > > > > > > > > > > > > an/netdisco-users/
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > > > >
> ==========================================
> > > > > > > > > > > > > > > > ======
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > > >
> ============================================
> > > > > > > > > > > > > > > ====
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > >
> ==============================================
> > > > > > > > > > > > > > ==
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > >
> ================================================
> > > > > > > > > > > > >
> _______________________________________________
> > > > > > > > > > > > > Netdisco mailing list
> > > > > > > > > > > > > [email protected]
> > > > > > > > > > > > > https://sourceforge.net/p/netdisco/mailman/netdi
> > > > > > > > > > > > > sco-users/
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > >
> ================================================
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > >
> ================================================
> > > > >
> > > > > --
> > > > > °((( = (( ===°°° (((
> > > > > ================================================
> > >
> > > --
> > > °((( = (( ===°°° (((
> > > ================================================
>
>
> _______________________________________________
> Netdisco mailing list
> [email protected]
> https://sourceforge.net/p/netdisco/mailman/netdisco-users/
>
> _______________________________________________
> Netdisco mailing list
> [email protected]
> https://sourceforge.net/p/netdisco/mailman/netdisco-users/
________________________________
Volg Aquafin op Facebook<https://www.facebook.com/AquafinNV> |
Twitter<https://twitter.com/aquafinnv> |
YouTube<http://www.youtube.com/channel/UCk_4P5BJ-MtEEDCkCsR_KqQ?feature=mhee> |
LinkedIN<http://www.linkedin.com/company/aquafin/products> |
Instagram<https://www.instagram.com/aquafin_nv/>
In het kader van de uitoefening van onze taken verzamelen we bij Aquafin
persoonsgegevens. Hoe we omgaan met deze gegevens en wat de rechten van de
betrokkenen zijn, kan je nalezen in onze privacy
policy<https://www.aquafin.be/nl-be/privacy-policy>.
P Denk aan het milieu. Druk deze mail niet onnodig af.
--- End Message ---
