--- Begin Message ---
Hi Oliver,
these are my current installed versions:
$ grep '$VERSION' /opt/netdisco/perl5/lib/perl5/App/Netdisco.pm
our $VERSION = '2.044014';
$ grep '$VERSION' /opt/netdisco/perl5/lib/perl5/MCE/Queue.pm
our $VERSION = '1.866';
I don’t see the MCS errors in the Netdisco-backend.log (any longer). Or do I
have to enable debugging?
However I see quite a few other job queue related(?) errors in the log (see
below).
The current situation for me is that the poller performance is nearly back to
normal.
But I still need to manually delete some left stucked jobs from the admin table
once at night via a cron job before the “Poller Performance” entries show up in
the Web UI.
Also I’m still facing the situation that after some time roughly half of the
configured pollers stopped for some reason. Re-starting the backend service
brings them all back for some hours.
Here is the netdisco-backend.log:
$ grep -i error ./logs/netdisco-backend.log* | grep PostgreSQL.pm | head -n 15
./logs/netdisco-backend.log:[120291] 2020-02-12 07:14:51 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[120413] 2020-02-12 07:15:21 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[126144] 2020-02-12 07:57:25 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[1158] 2020-02-12 08:38:02 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[1160] 2020-02-12 08:38:03 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[2026] 2020-02-12 08:45:41 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[3615] 2020-02-12 08:55:32 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[17774] 2020-02-12 10:15:03 error bless( {'msg' =>
'SQL::Abstract::puke(): [SQL::Abstract::__ANON__] Fatal: Operator calls in
update must be in the form { -op => $arg } at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267
./logs/netdisco-backend.log:[71090] 2020-02-13 07:01:59 error bless( {'msg' =>
'{UNKNOWN}: Can\'t call method "update" on an undefined value at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267. at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 274
./logs/netdisco-backend.log:[70923] 2020-02-13 07:02:00 error bless( {'msg' =>
'{UNKNOWN}: Can\'t call method "update" on an undefined value at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267. at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 274
./logs/netdisco-backend.log:[71128] 2020-02-13 07:02:00 error bless( {'msg' =>
'{UNKNOWN}: Can\'t call method "update" on an undefined value at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267. at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 274
./logs/netdisco-backend.log:[70982] 2020-02-13 07:02:00 error bless( {'msg' =>
'{UNKNOWN}: Can\'t call method "update" on an undefined value at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267. at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 274
./logs/netdisco-backend.log:[71087] 2020-02-13 07:02:01 error bless( {'msg' =>
'{UNKNOWN}: Can\'t call method "update" on an undefined value at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267. at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 274
./logs/netdisco-backend.log:[71096] 2020-02-13 07:02:03 error bless( {'msg' =>
'{UNKNOWN}: Can\'t call method "update" on an undefined value at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267. at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 274
./logs/netdisco-backend.log:[71075] 2020-02-13 07:02:03 error bless( {'msg' =>
'{UNKNOWN}: Can\'t call method "update" on an undefined value at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 267. at
/opt/netdisco/perl5/lib/perl5/App/Netdisco/JobQueue/PostgreSQL.pm line 274
netdisco@frdv02744:~ $
$ grep -i error ./logs/netdisco-backend.log* | grep PostgreSQL.pm | wc -l
299
Thanks,
Tobias
Von: Ricardo Stella <[email protected]>
Gesendet: Freitag, 14. Februar 2020 15:53
An: [email protected]
Betreff: Re: [Netdisco] scheduled jobs stop
I forgot to mention that not only jobs keep on running, but the log file shows
no errors since last startup.
On Thu, Feb 13, 2020 at 4:48 PM Oliver Gorwits
<[email protected]<mailto:[email protected]>> wrote:
Hi Gerlach
Please can you confirm this is still the case? Other users have reported the
same errors have stopped since the upgrade.
Can you check the version of MCE installed?
thanks
oliver.
On Mon, 10 Feb 2020 at 08:41, Gerlach Tobias DLN FIII31
<[email protected]<mailto:[email protected]>> wrote:
Hi Oliver,
Thank you for your efforts.
I’ve installed Netdisco 2.044014 yesterday but for me situation is unchanged,
workers stop, jobs stuck queued etc.
Regards,
Tobias
Von: Oliver Gorwits <[email protected]<mailto:[email protected]>>
Gesendet: Sonntag, 9. Februar 2020 11:05
An: [email protected]<mailto:[email protected]>
Cc:
[email protected]<mailto:[email protected]>
Betreff: Re: [Netdisco] scheduled jobs stop
Hello again
The developer of MCE found a bug, so I have released Netdisco 2.044014 which
will pull the new upstream MCE.
Please would you try that and let us know how it goes?
thanks
Oliver.
On Thu, 6 Feb 2020 at 08:54,
<[email protected]<mailto:[email protected]>> wrote:
Hi Oliver
2.0440013 is running
I'll inform you
Thank you
Marco
> Il 4 febbraio 2020 alle 22.36 Oliver Gorwits
> <[email protected]<mailto:[email protected]>> ha scritto:
>
> Hi,
> I have released Netdisco 2.044013 which uses a different configuration of the
> MCE job queue handler, on the advice of the MCE developer. This may or may
> not improve things (I've not been able to reproduce the bug), but it would be
> great if feedback comes soon, to let me know if this was a good move!
> many thanksOliver.
> On Tue, 4 Feb 2020 at 15:50, Ricardo Stella <
> [email protected]<mailto:[email protected]>> wrote:
> > This is happening on Redhat 7.7 but also on the old instance we were
> > migrating out from which is running Redhat 6.10.
> > On the older instance, we were running version 2.040006 since March 4 2019.
> > I Started the migration to a new instance (DB dump and import on new VM)
> > around January 17th this year.
> > On January 28, I upgraded the old instance to 2.044011 and started to see
> > the same problems as we are experiencing on the new VM. The newer version
> > of MCE:Queue is causing the problems?
> > Hope this helps - Ricardo.
> > On Tue, Feb 4, 2020 at 10:33 AM Oliver Gorwits <
> > [email protected]<mailto:[email protected]>> wrote:
> > > Can you report back with your operating systems, please?
> > > Many thanks,
> > > On Tue, 4 Feb 2020 at 15:20, Ricardo Stella <
> > > [email protected]<mailto:[email protected]>> wrote:
> > > > Thanks - same issues here. A couple of errors during the last 24 hours
> > > > since I restarted it as the queue was not doing anything over the
> > > > weekend. These are just a few:
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 25006.
> > > > Argument "" isn't numeric in read at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> > > > offset 1 of input at srl_decoder.c line 580 at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 30713.
> > > > Argument "" isn't numeric in read at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> > > > offset 1 of input at srl_decoder.c line 580 at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 48775.
> > > > Argument "" isn't numeric in read at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439, line 1.
> > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> > > > offset 1 of input at srl_decoder.c line 580 at
> > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1445, line 1.
> > > > Argument "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > isn't numeric in int at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > line 1484, line 49491.
> > > >
> > > >
> > > > On Tue, Feb 4, 2020 at 8:12 AM Oliver Gorwits <
> > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > Hi Marco
> > > > > I have emailed the developer of the MCE distribution to ask, as I
> > > > > think this is outside of Netdisco's domain,
> > > > > regardsOliver.
> > > > > On Tue, 4 Feb 2020 at 11:36, <
> > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > Hi Oliver
> > > > > >
> > > > > >
> > > > > > It stop again after 4 days
> > > > > >
> > > > > > [14758] 2020-01-30 11:14:30 debug [172.17.121.2] arpnip - processed
> > > > > > 0 IPv6 Neighbor Cache entries
> > > > > > [14758] 2020-01-30 11:14:30 info pol (3): wrapping up arpnip
> > > > > > job(22433622) - status done at Thu Jan 30 12:14:30 2020
> > > > > > Argument "PID_14758" isn't numeric in abs at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line
> > > > > > 128834.
> > > > > > Can't call method "_mce_m_pending" on an undefined value at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 679, line 128835.
> > > > > > [14900] 2020-01-30 13:50:47 warn App::Netdisco 2.044004 backend
> > > > > > [14900] 2020-01-30 13:50:47 info resolving backend hostname...
> > > > > > *************
> > > > > > [14904] 2020-02-03 19:13:41 info mgr (2): job 22463635 booked out
> > > > > > for this processing node
> > > > > > [14904] 2020-02-03 19:13:41 debug mgr (2): sleeping now...
> > > > > > Argument "_12455" isn't numeric in read at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 204, line
> > > > > > 476776.
> > > > > > Argument
> > > > > > "=rl^D\0A,{App::Netdisco::Backend::Job(*^Ok_statuslist@f..." isn't
> > > > > > numeric in abs at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line
> > > > > > 476776.
> > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal document. at
> > > > > > offset 1 of input at srl_decoder.c line 580 at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 480, line
> > > > > > 476779.
> > > > > > [12684] 2020-02-04 08:13:38 warn App::Netdisco 2.044004 backend
> > > > > >
> > > > > > It seems to occur randomly, but reading in the log I see that
> > > > > > Argument "SOMETHING" isn't numeric in read at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/SOMETHING.pm
> > > > > >
> > > > > > occur sometimes but usually don't zombies netdisco-backend
> > > > > >
> > > > > > it stop after that sequence
> > > > > > [14758] 2020-01-30 11:14:30 info pol (3): wrapping up arpnip
> > > > > > job(22433622) - status done at Thu Jan 30 12:14:30 2020
> > > > > > Argument "PID_14758" isn't numeric in abs at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 206, line
> > > > > > 128834.
> > > > > >
> > > > > > [12455] 2020-02-03 19:13:41 info pol (3): wrapping up arpnip
> > > > > > job(22463592) - status done at Mon Feb 3 20:13:41 2020
> > > > > > ...
> > > > > > Argument "_12455" isn't numeric in read at
> > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm line 204, line
> > > > > > 476776.
> > > > > >
> > > > > > just my two cents
> > > > > >
> > > > > > anyway can you suggest me how increase size of log?
> > > > > > cause in debugging mode the 7 files isn't enough for 2 days
> > > > > >
> > > > > > Thank you all
> > > > > > Marco
> > > > > >
> > > > > > > Il 30 gennaio 2020 alle 20.31 Oliver Gorwits <
> > > > > > > [email protected]<mailto:[email protected]>> ha scritto:
> > > > > > >
> > > > > > > I was looking to see if the issue is related to an upstream
> > > > > > > library change, rather than in Netdisco.
> > > > > > > Mainly because I'm scratching my head trying to work out what
> > > > > > > would cause this, and I can't yet reproduce it.
> > > > > > > On Wed, 29 Jan 2020 at 16:00, Ricardo Stella <
> > > > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > > > Almost there...
> > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv perl -MSereal\ 999 -e 1
> > > > > > > > Sereal version 999 required--this is only version 4.007.
> > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv perl -MMCE::Queue\ 999 -e
> > > > > > > > 1
> > > > > > > > MCE::Queue version 999 required--this is only version 1.865.
> > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > [netdisco@netdisco ~]$ ~/bin/localenv cpanm Sereal MCE
> > > > > > > > Sereal is up to date. (4.007)
> > > > > > > > MCE is up to date. (1.865)
> > > > > > > >
> > > > > > > > I assume we are trying to delete them and force download?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jan 29, 2020 at 10:52 AM Oliver Gorwits <
> > > > > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > > > > Sorry, my apologies, yes you would need to add "
> > > > > > > > > ~/bin/localenv" to the start of all those commands, I believe
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, 29 Jan 2020 at 15:17, Ricardo Stella <
> > > > > > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > > > > > Running as the netdisco user, I'm getting:
> > > > > > > > > > Can't locate Sereal.pm in @INC (@INC contains:
> > > > > > > > > > /usr/local/lib64/perl5 /usr/local/share/perl5
> > > > > > > > > > /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl
> > > > > > > > > > /usr/lib64/perl5 /usr/share/perl5 .).
> > > > > > > > > > BEGIN failed--compilation aborted.
> > > > > > > > > >
> > > > > > > > > > Does it need --local-lib ~/perl5 or ~/bin/localenv first?
> > > > > > > > > > And --notest?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Jan 29, 2020 at 9:47 AM Oliver Gorwits <
> > > > > > > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > > > > > > Hi Ricardo
> > > > > > > > > > > Please can you also run:perl -MSereal\ 999 -e 1perl
> > > > > > > > > > > -MMCE::Queue\ 999 -e 1
> > > > > > > > > > > Then runcpanm Sereal MCE
> > > > > > > > > > > and then let us know if the problem is still there?
> > > > > > > > > > > thanks,oliver.
> > > > > > > > > > > On Wed, 29 Jan 2020 at 14:15, Ricardo Stella <
> > > > > > > > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > > > > > > > Well, it's definitely a bug with the latest versions.
> > > > > > > > > > > > I upgraded the original instance I had which was
> > > > > > > > > > > > running fine under 2.040006 since March of last year.
> > > > > > > > > > > > This one also is exhibiting the same issues with jobs
> > > > > > > > > > > > queued since 5:30pm yesterday.
> > > > > > > > > > > > Error logs on that instance since last restart
> > > > > > > > > > > > yesterday afternoon are:
> > > > > > > > > > > > [7901] 2020-01-28 16:03:03 warn App::Netdisco 2.044011
> > > > > > > > > > > > backend
> > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439,
> > > > > > > > > > > > <$__ANONIO__> line 1.
> > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > > > > > > > document. at offset 1 of input at srl_decoder.c line
> > > > > > > > > > > > 580 at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > 1445, <$__ANONIO__> line 1.
> > > > > > > > > > > > Argument
> > > > > > > > > > > > "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > > > > > > > > > isn't numeric in int at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484,
> > > > > > > > > > > > <$__ANONIO__> line 1753.
> > > > > > > > > > > > Argument
> > > > > > > > > > > > "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > > > > > > > > > isn't numeric in int at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1484,
> > > > > > > > > > > > <$__ANONIO__> line 15984.
> > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439,
> > > > > > > > > > > > <$__ANONIO__> line 1.
> > > > > > > > > > > > Can't call method "status" without a package or object
> > > > > > > > > > > > reference at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/App/Netdisco/Backend/Role/Poller.pm
> > > > > > > > > > > > line 38, <$__ANONIO__> line 1.
> > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line 1439,
> > > > > > > > > > > > <$__ANONIO__> line 1.
> > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > > > > > > > document. at offset 1 of input at srl_decoder.c line
> > > > > > > > > > > > 580 at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > 1445, <$__ANONIO__> line 1.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jan 28, 2020 at 11:18 AM Ricardo Stella <
> > > > > > > > > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > > > > > > > > And just noticed that there's a newer version out
> > > > > > > > > > > > > there. Updated the new instance (including wiping the
> > > > > > > > > > > > > perl5 directory) and right after I started it, I got
> > > > > > > > > > > > > an error message. The old one was also updated but
> > > > > > > > > > > > > it's not giving me any errors so far.
> > > > > > > > > > > > > [8849] 2020-01-28 16:13:41 warn App::Netdisco
> > > > > > > > > > > > > 2.044011 backend
> > > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > 1439, line 1.
> > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid Sereal
> > > > > > > > > > > > > document. at offset 1 of input at srl_decoder.c line
> > > > > > > > > > > > > 580 at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > line 1445, line 1.
> > > > > > > > > > > > > Argument
> > > > > > > > > > > > > "=M-srl^D\0A,{App::Netdisco::Backend::Job(*^Ofstatusfqueu..."
> > > > > > > > > > > > > isn't numeric in int at
> > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > 1484, line 32.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jan 28, 2020 at 9:56 AM Ricardo Stella <
> > > > > > > > > > > > > [email protected]<mailto:[email protected]>> wrote:
> > > > > > > > > > > > > > Same here...
> > > > > > > > > > > > > > backend status thinks it's running but jobs are
> > > > > > > > > > > > > > queued since last night and not running. Here are
> > > > > > > > > > > > > > the errors since last restart yesterday:
> > > > > > > > > > > > > > [24657] 2020-01-27 16:00:58 warn App::Netdisco
> > > > > > > > > > > > > > 2.044009 backend
> > > > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > 1439, line 1.
> > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid
> > > > > > > > > > > > > > Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > 1445, line 1.
> > > > > > > > > > > > > > Argument "=M-srl^D\0A,n yesterday after few hour.
> > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > [5754] 2020-01-27 17:06:59 debug -> run worker
> > > > > > > > > > > > > > main/wirelessnodes/100
> > > > > > > > > > > > > > [5754] 2020-01-27 17:06:59 info pol (3):
> > > > > > > > > > > > > > wrapping up macsuck job(22425208) - status done at
> > > > > > > > > > > > > > Mon Jan 27 18:06:59 2020
> > > > > > > > > > > > > > [5750] 2020-01-27 17:06:59 debug
> > > > > > > > > > > > > > [172.17.119.6] macsuck - port 1:43 vlan unknown : 1
> > > > > > > > > > > > > > nodes
> > > > > > > > > > > > > > Argument "PID_5754" isn't numeric in abs at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Core/Manager.pm
> > > > > > > > > > > > > > line 206, line 32948.
> > > > > > > > > > > > > > Can't call method "_mce_m_pending" on an
> > > > > > > > > > > > > > undefined value at
> > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > 679, line 32949.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I activated debug, it seems that some scheduled
> > > > > > > > > > > > > > jobs (macsuck, discoverall etc.) cause the error
> > > > > > > > > > > > > > "Argument "PID_####" isn't numeric " and it zombies
> > > > > > > > > > > > > > netdisco-backend child
> > > > > > > > > > > > > > ps aux | grep netd
> > > > > > > > > > > > > > netdisco 3428 0.0 0.3 22840 15848 ?
> > > > > > > > > > > > > > S gen27 2:05 netdisco-backend
> > > > > > > > > > > > > > netdisco 3429 0.0 0.0 0 0 ?
> > > > > > > > > > > > > > Z gen27 0:15 [nd2: master]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I can't say if it is caused by my new
> > > > > > > > > > > > > > setup/configuration or something else
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Marco
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Il 27 gennaio 2020 alle 17.03 Ricardo Stella <
> > > > > > > > > > > > > > > [email protected]<mailto:[email protected]>> ha
> > > > > > > > > > > > > > > scritto:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Also happening here. I also had exported the DB
> > > > > > > > > > > > > > > in order to install on a new VM with new OS. Had
> > > > > > > > > > > > > > > a couple of problems that I posted but had this
> > > > > > > > > > > > > > > same error on the logs.
> > > > > > > > > > > > > > > Noticed all jobs queued for a couple of days and
> > > > > > > > > > > > > > > nothing running.
> > > > > > > > > > > > > > > Last message on logs was:
> > > > > > > > > > > > > > > Argument "" isn't numeric in read at
> > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > > 1439, line 1.
> > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a valid
> > > > > > > > > > > > > > > Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm line
> > > > > > > > > > > > > > > 1445, line 1.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Restarting it seems to get the jobs running again.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Mon, Jan 27, 2020 at 10:54 AM marco via
> > > > > > > > > > > > > > > netdisco-users <
> > > > > > > > > > > > > > > [email protected]<mailto:[email protected]>>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > Hi there
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I had set up a new ND2 host on debian buster
> > > > > > > > > > > > > > > > some weeks ago
> > > > > > > > > > > > > > > > for experimental purpose
> > > > > > > > > > > > > > > > I have another ND2 host up and running since
> > > > > > > > > > > > > > > > years
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Software Version
> > > > > > > > > > > > > > > > App::Netdisco 2.44.4
> > > > > > > > > > > > > > > > SNMP::Info 3.70
> > > > > > > > > > > > > > > > DB Schema 61
> > > > > > > > > > > > > > > > PostgreSQL 12.00.1
> > > > > > > > > > > > > > > > Perl 5.28.1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I restore db from another ND2
> > > > > > > > > > > > > > > > and copy deployment.yml
> > > > > > > > > > > > > > > > It worked
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > But I noticed that it stops running the
> > > > > > > > > > > > > > > > scheduled jobs after some times (days)
> > > > > > > > > > > > > > > > I had to restart netdisco-backend,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > here some info I collect
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > from netdisco-backend.log
> > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 debug mgr (2):
> > > > > > > > > > > > > > > > getting potential jobs for 1 workers
> > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 debug
> > > > > > > > > > > > > > > > [172.17.185.50] arpnip - processed 373 ARP
> > > > > > > > > > > > > > > > Cache entries
> > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 debug
> > > > > > > > > > > > > > > > [172.17.185.50] arpnip - processed 0 IPv6
> > > > > > > > > > > > > > > > Neighbor Cache entries
> > > > > > > > > > > > > > > > [2700] 2020-01-24 15:15:18 info pol (3):
> > > > > > > > > > > > > > > > wrapping up arpnip job(22423168) - status done
> > > > > > > > > > > > > > > > at Fri Jan 24 16:15:18 2020
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 debug getsome:
> > > > > > > > > > > > > > > > cancelled 0E0 duplicate(s) of job 22423235
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 info mgr (2):
> > > > > > > > > > > > > > > > job 22423235 booked out for this processing node
> > > > > > > > > > > > > > > > Argument "PID_2700" isn't numeric in read
> > > > > > > > > > > > > > > > at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 477, line 31470.
> > > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a
> > > > > > > > > > > > > > > > valid Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 480, line 31470.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > root@deb-netdisco:~# systemctl status
> > > > > > > > > > > > > > > > netdisco-backend.service
> > > > > > > > > > > > > > > > ● netdisco-backend.service - Netdisco
> > > > > > > > > > > > > > > > Backend Service
> > > > > > > > > > > > > > > > Loaded: loaded
> > > > > > > > > > > > > > > > (/etc/systemd/system/netdisco-backend.service;
> > > > > > > > > > > > > > > > enabled; vendor preset: enabled)
> > > > > > > > > > > > > > > > Active: active (running) since Fri
> > > > > > > > > > > > > > > > 2020-01-24 09:53:03 CET; 3 days ago
> > > > > > > > > > > > > > > > Process: 110
> > > > > > > > > > > > > > > > ExecStart=/home/netdisco/bin/netdisco-backend
> > > > > > > > > > > > > > > > start (code=exited, status=0/SUCCESS)
> > > > > > > > > > > > > > > > Main PID: 216 (netdisco-backen)
> > > > > > > > > > > > > > > > Tasks: 2 (limit: 4915)
> > > > > > > > > > > > > > > > Memory: 143.0M
> > > > > > > > > > > > > > > > CGroup:
> > > > > > > > > > > > > > > > /system.slice/netdisco-backend.service
> > > > > > > > > > > > > > > > └─216 netdisco-backend
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > gen 24 09:53:02 deb-netdisco systemd[1]:
> > > > > > > > > > > > > > > > Starting Netdisco Backend Service...
> > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: Netdisco Backend
> > > > > > > > > > > > > > > > [Started]
> > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: config watcher: watching
> > > > > > > > > > > > > > > > /home/netdisco/environments for updates.
> > > > > > > > > > > > > > > > gen 24 09:53:03 deb-netdisco systemd[1]:
> > > > > > > > > > > > > > > > Started Netdisco Backend Service.
> > > > > > > > > > > > > > > > gen 24 10:01:48 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: --
> > > > > > > > > > > > > > > > /home/netdisco/environments/deployment.yml
> > > > > > > > > > > > > > > > updated.
> > > > > > > > > > > > > > > > gen 24 10:01:48 deb-netdisco
> > > > > > > > > > > > > > > > netdisco-backend[110]: config watcher: sending
> > > > > > > > > > > > > > > > TERM to the server (pid:217)...
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > root@deb-netdisco:~# ps aux | grep netd
> > > > > > > > > > > > > > > > netdisco 216 0.0 0.3 22840 16008 ?
> > > > > > > > > > > > > > > > S gen24 6:19 netdisco-backend
> > > > > > > > > > > > > > > > netdisco 281 0.0 0.3 20744 13680 ?
> > > > > > > > > > > > > > > > S gen24 0:00 perl
> > > > > > > > > > > > > > > > /home/netdisco/bin/netdisco-web start
> > > > > > > > > > > > > > > > netdisco 282 0.0 0.3 22152 16696 ?
> > > > > > > > > > > > > > > > S gen24 0:47 starman master
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 372 0.0 0.0 0 0 ?
> > > > > > > > > > > > > > > > Z gen24 0:16 [nd2: master]
> > > > > > > > > > > > > > > > netdisco 373 0.0 2.7 135148 117200 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 374 0.0 2.8 136000 118000 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 375 0.0 2.7 133744 115940 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 376 0.0 2.8 137420 119504 ?
> > > > > > > > > > > > > > > > S gen24 0:06 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > netdisco 377 0.0 2.7 133792 115996 ?
> > > > > > > > > > > > > > > > S gen24 0:05 starman worker
> > > > > > > > > > > > > > > > --disable-keepalive --user 1001 --group 1001
> > > > > > > > > > > > > > > > /home/netdisco/perl5/bin/netdisco-web-fg
> > > > > > > > > > > > > > > > root 3405 0.0 0.0 6096 824 pts/0
> > > > > > > > > > > > > > > > S+ 10:59 0:00 grep netd
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > after stop and start
> > > > > > > > > > > > > > > > root@deb-netdisco:~# systemctl start
> > > > > > > > > > > > > > > > netdisco-backend.service
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > it seems to work again
> > > > > > > > > > > > > > > > [392] 2020-01-24 15:15:18 info mgr (2):
> > > > > > > > > > > > > > > > job 22423235 booked out for this processing node
> > > > > > > > > > > > > > > > Argument "PID_2700" isn't numeric in read
> > > > > > > > > > > > > > > > at /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 477, line 31470.
> > > > > > > > > > > > > > > > Sereal: Error: Bad Sereal header: Not a
> > > > > > > > > > > > > > > > valid Sereal document. at offset 1 of input at
> > > > > > > > > > > > > > > > srl_decoder.c line 580 at
> > > > > > > > > > > > > > > > /home/netdisco/perl5/lib/perl5/MCE/Queue.pm
> > > > > > > > > > > > > > > > line 480, line 31470.
> > > > > > > > > > > > > > > > [3429] 2020-01-27 10:10:08 warn
> > > > > > > > > > > > > > > > App::Netdisco 2.044004 backend
> > > > > > > > > > > > > > > > [3429] 2020-01-27 10:10:08 info resolving
> > > > > > > > > > > > > > > > backend hostname...
> > > > > > > > > > > > > > > > [3433] 2020-01-27 10:10:08 info applying
> > > > > > > > > > > > > > > > role Scheduler to worker 1
> > > > > > > > > > > > > > > > [3436] 2020-01-27 10:10:08 info applying
> > > > > > > > > > > > > > > > role Poller to worker 4
> > > > > > > > > > > > > > > > ...
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > > > > > Netdisco mailing list
> > > > > > > > > > > > > > > > [email protected]<mailto:[email protected]>
> > > > > > > > > > > > > > > > https://sourceforge.net/p/netdisco/mailman/netdisco-users/
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > > > ================================================
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > > ================================================
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > > ================================================
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > > > ================================================
> > > > > > > > > > > > _______________________________________________
> > > > > > > > > > > > Netdisco mailing list
> > > > > > > > > > > > [email protected]<mailto:[email protected]>
> > > > > > > > > > > > https://sourceforge.net/p/netdisco/mailman/netdisco-users/
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > °((( = (( ===°°° (((
> > > > > > > > > > ================================================
> > > > > > > >
> > > > > > > > --
> > > > > > > > °((( = (( ===°°° (((
> > > > > > > > ================================================
> > > >
> > > > --
> > > > °((( = (( ===°°° ((( ================================================
> >
> > --
> > °((( = (( ===°°° ((( ================================================
_______________________________________________
Netdisco mailing list
[email protected]<mailto:[email protected]>
https://sourceforge.net/p/netdisco/mailman/netdisco-users/
--
°(((=((===°°°(((================================================
--- End Message ---
