WorkerLog is reduced now thanks.
Re the logging, something further in the file is changing this.
If I run grep against the file for something simple like the character 1 it
outputs hits all the way down to:
2016-10-06 06:19:17 [Worker_1] Worker_1 wakes up
It then stops and outputs "binary file matches" as it sees characters in
the stream that aren't consistent with the file type.
Are "bad line-end sequence" and "unsupported_SMTP_line_terminator" things
produced by the higher WorkerLog setting and will they go away now it is
set back to standard or do I have to change something else to suppress them?
On Thu, Oct 6, 2016 at 9:50 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:
> >I'm wondering if the Ubuntu implementation just starts at the number 70
> for some reason.
>
> This is not possible. Global filehandle numbers are starting at 0, which
> is STDOUT on every OS.
> Don't care about as long as the process runs well.
>
> >I believe this is caused by ASSP outputting some raw SMTP data to the
> logfile.
>
> maillog.txt is an UTF-8 file (contains the UTF-8 BOM at its start).
>
> If there everyting is running well, reduce the WorkerLog setting.
>
> Thomas
>
>
>
> Von: cw <colin.war...@gmail.com>
> An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> Datum: 06.10.2016 10:38
> Betreff: Re: [Assp-test] unable to detect any running worker
>
>
>
> Hi Thomas,
>
> Thank you very much for your help and suggestions with this.
>
> So far both servers are running without shut down since I put 79_1 on
> yesterday.
>
> I am still seeing lots of can't interrupt worker - for example at 04:48:35
> when the server wasn't doing much I saw 70 lines logged within the same
> second. One server had a total of 3,500 logs between midnight and 9am.
>
> I'm also seeing issues with grep treating the maillog files as binary
> which
> I haven't seen before build 16270.
>
> I believe this is caused by ASSP outputting some raw SMTP data to the
> logfile. For example I see a load of none printable characters immediately
> before the timestamp on a line that says 2016-10-06 07:01:16 [Worker_2]
> [unsupported_SMTP_line_terminator]
>
> Similarly I also see lines such as:
>
> 2016-10-06 07:01:29 [Worker_2] 5.10.252.206 info: bad line-end sequence in
> ###
>
> Where ### is another load of none-printable characters.
>
> I presume this is someone running an exploit scanner as the IP address
> traces to Azerbaijan.
>
> All my logging is to standard at the moment except WorkerLog, is there any
> way to have this information limited to the connection debug files so it
> doesn't make grep treat the maillog as a binary file?
>
> I had a think on the number of perl file handles. I can't explain it as
> all
> these servers do is run ASSP and exim4. I have a couple of nightly shell
> scripts but nothing that would run for more than a few seconds. ps only
> ever shows the ASSP perl process. I'm wondering if the Ubuntu
> implementation just starts at the number 70 for some reason.
>
> On Wed, Oct 5, 2016 at 5:52 PM, Thomas Eckardt
> <thomas.ecka...@thockar.com>
> wrote:
>
> > 2.5.4 16279_1 is available at CVS /test.
> >
> > I tried to optimize the worker lookup process.
> >
> > Thomas
> >
> >
> >
> >
> >
> > Von: cw <colin.war...@gmail.com>
> > An: ASSP development mailing list <assp-test@lists.sourceforge.net>
> > Datum: 05.10.2016 18:08
> > Betreff: Re: [Assp-test] unable to detect any running worker
> >
> >
> >
> > So actually a development.
> >
> > It looks like disabling EnableHighPerformance has lightened the load a
> > little. I've gone an hour without a complete failure and it explains why
> > I'm now seeing "Main_Thread can't interrupt Worker" instead of "unable
> to
> > detect any running worker".
> >
> > Presumably with high performance set to off ASSP has enough time to
> > recover
> > when the issue occurs. Hopefully this will buy me some leeway from
> having
> > to find an alternative and find the root cause of the issue.
> >
> > On Wed, Oct 5, 2016 at 4:40 PM, cw <colin.war...@gmail.com> wrote:
> >
> > > Unfortunately not. Most of the time the GUI is so unresponsive that I
> > only
> > > go into it when necessary. I understand the performance impact on it.
> > >
> > > Everything works well when ASSP starts, but within a few minutes it
> all
> > > starts to slow down and become unresponsive. I have a feeling that it
> is
> > > either going to be something really simple or some odd combination of
> > > events that I have been unfortunate enough to end up with.
> > >
> > > Most of the time I use command line and tail/grep/vi. Incidentally
> since
> > > yesterday I'm getting issues with grep reporting binary file matches
> on
> > the
> > > logs instead of outputting the results. I can only presume some of the
> > > extra debug information is fooling grep but it doesn't help trying to
> > track
> > > down the right lights in the log!
> > >
> > > On Wed, Oct 5, 2016 at 4:32 PM, Thomas Eckardt
> > <thomas.ecka...@thockar.com
> > > > wrote:
> > >
> > >> >2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by
> > interrupted
> > >> >Worker_4 in 10.758 seconds - got (ok)
> > >>
> > >> I'm looking for the reason of this very huge time value of 10.758
> > seconds.
> > >>
> > >> The only reason I can find is - you are running constantly at least
> one
> > >> auto refreshing GUI screen. I'm right?
> > >>
> > >> switch to 'tail' the maillog.txt on a console session
> > >>
> > >> tail -fn 30 /usr...../maillog.txt
> > >>
> > >> do not use the shutdown_list (connection screen)!
> > >>
> > >> How it works now?
> > >>
> > >> Thomas
> > >>
> > >>
> > >>
> > >> Von: cw <colin.war...@gmail.com>
> > >> An: ASSP development mailing list
> <assp-test@lists.sourceforge.net>
> > >> Datum: 05.10.2016 17:05
> > >> Betreff: Re: [Assp-test] unable to detect any running worker
> > >>
> > >>
> > >>
> > >> I specifically checked netstat for established connections and didn't
> > >> include any in the wait state or part open state. According to the OS
> > >> these
> > >> are connections still fully open. I excluded the ones to the MTA so
> > this
> > >> should be purely inbound SMTP connections.
> > >>
> > >> I've set EnableHighPerformance back to off.
> > >>
> > >> Loads of runs produced:
> > >>
> > >> 0.003068
> > >> 0.003083
> > >> 0.003115
> > >> 0.003108
> > >> 0.003101
> > >> 0.003095
> > >> 0.0031
> > >> 0.003096
> > >> 0.003096
> > >> 0.003096
> > >> 0.003105
> > >> 0.003099
> > >> 0.003097
> > >> 0.003095
> > >> 0.00308
> > >> 0.003071
> > >> 0.003071
> > >> 0.003083
> > >> 0.003077
> > >> 0.003072
> > >> 0.003076
> > >> 0.003075
> > >> 0.003075
> > >> 0.003072
> > >> 0.0031
> > >> 0.003077
> > >> 0.003094
> > >> 0.003099
> > >>
> > >>
> > >> On Wed, Oct 5, 2016 at 3:45 PM, Thomas Eckardt
> > >> <thomas.ecka...@thockar.com>
> > >> wrote:
> > >>
> > >> > >I'm currently comparing things. shutdown_list shows 5 smtp
> sessions
> > in
> > >> > >threads whilst netstat shows 18 connections established to port 25
> > from
> > >> > the
> > >> > >outside world.
> > >> > >It then went down to 2 SMTP sessions in threads and netstat showed
> > 13
> > >> > >connections.
> > >> >
> > >> > the state of most of the connection should than be something like
> > >> > "fin_wait". Closed connection are keeped by the OS socket handler
> for
> > >> two
> > >> > minutes to satisfy the peer, if there are still pakets on the way.
> > >> >
> > >> > set 'EnableHighPerformance' back to the default - 'OFF'
> > >> >
> > >> >
> > >> > please run the following small perl script and tell me the output,
> > >> > Possibly run it multiple times. Is the sleep time very different?
> > >> > I expect to see a sleep time of 0.003 to 0.005 seconds
> > >> >
> > >> > use strict;
> > >> > use Time::HiRes;
> > >> >
> > >> > my $t = 3000/1000000;
> > >> > print Time::HiRes::sleep($t);
> > >> >
> > >> > Thomas
> > >> >
> > >> >
> > >> > Von: cw <colin.war...@gmail.com>
> > >> > An: ASSP development mailing list
> > <assp-test@lists.sourceforge.net>
> > >> > Datum: 05.10.2016 16:27
> > >> > Betreff: Re: [Assp-test] unable to detect any running worker
> > >> >
> > >> >
> > >> >
> > >> > ThreadCycleTime is set to 3000 microseconds
> > >> > EnableHighPerformance I have changed during the course of this week
> > with
> > >> > no
> > >> > effect on the problem. It was at default and is now at the highest
> > >> > setting.
> > >> > I'd check the exact names for the settings except neither server
> will
> > >> give
> > >> > me a connection to the GUI at the moment.
> > >> >
> > >> > I'm currently comparing things. shutdown_list shows 5 smtp sessions
> > in
> > >> > threads whilst netstat shows 18 connections established to port 25
> > from
> > >> > the
> > >> > outside world.
> > >> > It then went down to 2 SMTP sessions in threads and netstat showed
> 13
> > >> > connections.
> > >> >
> > >> > On Wed, Oct 5, 2016 at 3:06 PM, Thomas Eckardt
> > >> > <thomas.ecka...@thockar.com>
> > >> > wrote:
> > >> >
> > >> > > What are your setting for :
> > >> > >
> > >> > > ThreadCycleTime
> > >> > > EnableHighPerformance
> > >> > >
> > >> > > Thomas
> > >> > >
> > >> > >
> > >> > >
> > >> > > Von: cw <colin.war...@gmail.com>
> > >> > > An: ASSP development mailing list
> > >> <assp-test@lists.sourceforge.net>
> > >> > > Datum: 05.10.2016 15:51
> > >> > > Betreff: Re: [Assp-test] unable to detect any running
> worker
> > >> > >
> > >> > >
> > >> > >
> > >> > > I can confirm that this number is artificially high quite easily.
> > >> > > I actually have a third ASSP instance. This one is specially
> > >> configured
> > >> > to
> > >> > > only accept emails from Office 365 and acts as an outbound relay
> > >> purely
> > >> > to
> > >> > > gather messages for the corpus. This one does not experience the
> > >> problem
> > >> > > with no running workers being available.
> > >> > >
> > >> > > Its stats haven't been reset for a long time and show an average
> of
> > >> 2901
> > >> > > per day. Concurrent SMTP sessions since last restart are 0 (4
> max).
> > >> > >
> > >> > > The last message showing in the logs says:
> > >> > >
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread got
> connection
> > >> > request
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread looks up the
> > best
> > >> > > Worker for new connection - 75
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread will wait
> (max
> > 30
> > >> s)
> > >> > > for the answer of Worker_7 which handles 0 sockets
> > >> > > 2016-10-05 14:43:57 [Worker_7] Worker_7 wakes up
> > >> > > 2016-10-05 14:43:57 [Main_Thread] Info: Main_Thread freed by idle
> > >> > Worker_7
> > >> > > in 0.005 seconds - got (ok)
> > >> > > 2016-10-05 14:43:57 [Worker_7] Info: Worker_7 got connection from
> > >> > > MainThread - 75/75
> > >> > > 2016-10-05 14:43:57 [Worker_7] Info: Worker_7 freed Main_Thread -
> > 84
> > >> > >
> > >> > > There are no other emails going on at this time. Just the one
> > >> connection
> > >> > > and it still says 75. This server is firewalled and the only
> > messages
> > >> it
> > >> > > ever sees are from known Office 365 IP addresses along with the
> > >> internal
> > >> > > test server that sends 1 message every 60 seconds.
> > >> > >
> > >> > > On Wed, Oct 5, 2016 at 2:33 PM, cw <colin.war...@gmail.com>
> wrote:
> > >> > >
> > >> > > > Hi Thomas,
> > >> > > >
> > >> > > > Thank you for the rundown. The load is exceptionally high at
> the
> > >> > moment
> > >> > > > because everything backs up every time the mail servers stop.
> > Having
> > >> > > said
> > >> > > > that, I've just watched it happen again on one of the servers.
> It
> > >> only
> > >> > > had
> > >> > > > 7 messages showing in the shutdown_list when it happened.
> Looking
> > at
> > >> > the
> > >> > > > logs I see:
> > >> > > >
> > >> > > > 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 got connection
> from
> > >> > > > MainThread - 78/78
> > >> > > > 2016-10-05 14:23:50 [Main_Thread] Info: Main_Thread freed by
> > >> > interrupted
> > >> > > > Worker_4 in 10.758 seconds - got (ok)
> > >> > > > 2016-10-05 14:23:50 [Worker_4] Info: Worker_4 freed Main_Thread
> -
> > >> 101
> > >> > > >
> > >> > > > So something is artificially inflating the connection count. My
> > best
> > >> > > guess
> > >> > > > ties in with something else I was looking at where a remote
> > server
> > >> was
> > >> > > > opening two connections that never got answered. It only got
> > through
> > >> > on
> > >> > > the
> > >> > > > third connection and that could be replicated each time. I put
> it
> > >> down
> > >> > > to
> > >> > > > some odd firewall on their end but maybe I was wrong in doing
> > that.
> > >> > > >
> > >> > > > The actual stats on one mailserver are:
> > >> > > >
> > >> > > > Last reset 8 days, 0 hours, 6 mins
> > >> > > > Messages process 165413 (20664 per day)
> > >> > > > SMTP Connections received 113278
> > >> > > >
> > >> > > > The other is:
> > >> > > > Last reset 8 days, 0 hours, 18 mins
> > >> > > > Messaged processed 131720 (16438 per day)
> > >> > > > SMTP Connections received 93088
> > >> > > >
> > >> > > > The stats reset themselves during on of the upgrades of ASSP,
> not
> > >> sure
> > >> > > why.
> > >> > > >
> > >> > > > Each server is 16 CPU cores, 12GB RAM. DB is on a separate
> MySQL
> > box
> > >> > > > though MTA and Clam run local. URIBL is disabled. I don't use
> > OCR,
> > >> DCC
> > >> > > or
> > >> > > > Razor. I tried to get OCR working once and all hell broke lose
> > and I
> > >> > > > haven't been back since.
> > >> > > >
> > >> > > > I've been monitoring the performance of the servers all along
> and
> > >> not
> > >> > > > seeing any bottlenecks in CPU, RAM or network.When the problem
> > >> started
> > >> > > the
> > >> > > > mail servers had 8 and 10GB RAM respectively (10GB for the one
> > >> running
> > >> > > > rebuild).
> > >> > > > All the best,
> > >> > > > Colin
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Oct 5, 2016 at 2:07 PM, Thomas Eckardt
> > >> > > <thomas.ecka...@thockar.com
> > >> > > > > wrote:
> > >> > > >
> > >> > > >> It seems, that this system is overloaded with active
> > connections.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by
> > >> > > interrupted
> > >> > > >> Worker_3 in 2.972 seconds - got (ok)
> > >> > > >>
> > >> > > >> Even this time value is ten times higher than expected.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed
> Main_Thread
> > -
> > >> > 170
> > >> > > >>
> > >> > > >> The '170' is the filehandle number in perl (counted from
> 1.....)
> > -
> > >> > this
> > >> > > >> implies 70 or more active connections.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait
> > (max
> > >> > 30
> > >> > > s)
> > >> > > >> for the answer of Worker_3 which handles 12 sockets
> > >> > > >>
> > >> > > >> This implies the same, if the equal balancing of all
> connections
> > >> over
> > >> > > all
> > >> > > >> seven workers works correctly.
> > >> > > >>
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> > >> > > >> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.103252172470093
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > >> 0.0278699398040771
> > >> > > >> >2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > >> 0.0423040390014648
> > >> > > >>
> > >> > > >> The processing time itself seems to be normal. SC-Time - is
> the
> > >> time
> > >> > > >> required to process one loop (read + queue or read + check +
> > queue)
> > >> > for
> > >> > > >> one active connection.
> > >> > > >>
> > >> > > >> Assuming an average of 4 second life time per connection - the
> > >> > workload
> > >> > > >> per day would be 1.5 million (70 / 4 *3600 *24) connections
> > based
> > >> on
> > >> > > the
> > >> > > >> provided output.
> > >> > > >> Configured to an ISP mode the avg. maximum for a single assp
> > >> instance
> > >> > > is
> > >> > > >> 800.000 connections. To be able to handle workload peeks, the
> > >> > > >> configuration should not allow more than 400.000 connection a
> > day.
> > >> > > >>
> > >> > > >> ISP mode:
> > >> > > >> - 16 CPU cores
> > >> > > >> - 16 Workers
> > >> > > >> - 24 GB RAM
> > >> > > >> - dedicated systems for MTA, enterprise database, ClamAV
> > >> > > >> - HMM and spamDB holded unshared in RAM
> > >> > > >> - high performance DNS servers
> > >> > > >> - limited DNS usage (disable most extensive DNS-using checks -
> > eg.
> > >> > > URIBL)
> > >> > > >> - no OCR
> > >> > > >> - no DCC
> > >> > > >> - no Razor2
> > >> > > >>
> > >> > > >> >2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by
> > >> > > interrupted
> > >> > > >> Worker_3 in 31.940 seconds - got (ok)
> > >> > > >>
> > >> > > >> This time value is near the end of the line. At this time a
> > small
> > >> > count
> > >> > > of
> > >> > > >> additionally connections can lead in to an assp shutdown,
> > because
> > >> the
> > >> > > >> workers don't accept new connections.
> > >> > > >>
> > >> > > >> Thomas
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> Von: cw <colin.war...@gmail.com>
> > >> > > >> An: ASSP development mailing list
> > >> > <assp-test@lists.sourceforge.net>
> > >> > > >> Datum: 05.10.2016 13:15
> > >> > > >> Betreff: Re: [Assp-test] unable to detect any running
> > worker
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> Thanks.
> > >> > > >>
> > >> > > >> I've had both servers come up against unable to detect any
> > running
> > >> > > worker
> > >> > > >> since clearing out all the files suggested. So I'm getting
> 16279
> > >> > > running
> > >> > > >> now.
> > >> > > >>
> > >> > > >> I noticed the startup with those files removed was really
> quick,
> > >> > > starting
> > >> > > >> back up the second time took several minutes so presumably
> > reading
> > >> > > those
> > >> > > >> files during startup takes a little while versus creating them
> > >> fresh.
> > >> > > >>
> > >> > > >> I've caught one already and traced it through the new logs:
> > >> > > >>
> > >> > > >> 016-10-05 11:47:38 [Main_Thread] Info: Main_Thread got
> > connection
> > >> > > request
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread looks up
> the
> > >> best
> > >> > > >> Worker for new connection - 73
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: try to interrupt
> worker
> > >> > > Worker_3
> > >> > > >> (12) for new connection
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread
> interrupted
> > >> > > Worker_3
> > >> > > >> (12) to submit the connection
> > >> > > >> 2016-10-05 11:47:38 [Main_Thread] Info: Main_Thread will wait
> > (max
> > >> 30
> > >> > > s)
> > >> > > >> for the answer of Worker_3 which handles 12 sockets
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] SC-Time Worker_3:
> > 0.0462169647216797
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 got connection
> > from
> > >> > > >> MainThread - 73/73
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] Info: Worker_3 freed
> Main_Thread
> > -
> > >> 170
> > >> > > >> 2016-10-05 11:47:41 [Main_Thread] Info: Main_Thread freed by
> > >> > > interrupted
> > >> > > >> Worker_3 in 2.972 seconds - got (ok)
> > >> > > >> 2016-10-05 11:47:41 [Worker_3] Connected: session:7FF3AA6E9A28
> > >> > > >> 109.168.50.75:41612 > 92.63.138.65:25 > 127.0.0.1:125
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.103252172470093
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.0278699398040771
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 [SMTP Reply] 220
> > >> > > >> mail2.smtphost.co.uk ESMTP Exim 4.86_2 Ubuntu Wed, 05 Oct 2016
> > >> > 11:47:41
> > >> > > >> +0100
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] SC-Time Worker_3:
> > 0.0423040390014648
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 SC-Time Worker_3:
> > >> > > >> 0.0296478271484375
> > >> > > >> 2016-10-05 11:47:43 [Worker_3] 109.168.50.75 disconnected:
> > >> > > >> session:7FF3AA6E9A28 109.168.50.75 - processing time 2 seconds
> > >> > > >> 2016-10-05 11:48:13 [Main_Thread] Info: Main_Thread freed by
> > >> > > interrupted
> > >> > > >> Worker_3 in 31.940 seconds - got (ok)
> > >> > > >>
> > >> > > >> In this case, it looks like the connection ended without any
> > actual
> > >> > > data.
> > >> > > >> There is lots of activity reported by Worker_3 in between
> > 11:47:43
> > >> > and
> > >> > > >> 11:48:13 but this all pertains to other connections that were
> > >> already
> > >> > > in
> > >> > > >> progress.
> > >> > > >>
> > >> > > >> That is a bit different to the earlier one where a message was
> > >> > received
> > >> > > >> and
> > >> > > >> the message completed within the 30s window.
> > >> > > >>
> > >> > > >> I'm not seeing anything to help me figure out why though and I
> > >> don't
> > >> > > want
> > >> > > >> to simply post a big excerpt of the maillog.txt.
> > >> > > >>
> > >> > > >> On Wed, Oct 5, 2016 at 11:02 AM, Thomas Eckardt
> > >> > > >> <thomas.ecka...@thockar.com>
> > >> > > >> wrote:
> > >> > > >>
> > >> > > >> > >I looked at SF but only see 16275 in test (updated 3 days
> > ago)
> > >> so
> > >> > > maybe
> > >> > > >> > it
> > >> > > >> > hasn't made its way live yet.
> > >> > > >> >
> > >> > > >> > Sorry, my background CVS sync was not running - update is
> > done.
> > >> > > >> >
> > >> > > >> > Thomas
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > Von: cw <colin.war...@gmail.com>
> > >> > > >> > An: ASSP development mailing list
> > >> > > <assp-test@lists.sourceforge.net>
> > >> > > >> > Datum: 05.10.2016 11:48
> > >> > > >> > Betreff: Re: [Assp-test] unable to detect any running
> > >> worker
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > Thank you Thomas.
> > >> > > >> >
> > >> > > >> > useDB4IntCache - already set to off
> > >> > > >> > I've set WorkerLog to diagnostic and done the other steps.
> > >> > > >> > I don't have anything in CorrectASSPcfg.pm so as part of the
> > >> > > >> > troubleshooting I have previously deleted it and downloaded
> a
> > >> fresh
> > >> > > copy
> > >> > > >> > from SourceForge.
> > >> > > >> >
> > >> > > >> > I have to say that ASSP started up within seconds after
> > clearing
> > >> > > those
> > >> > > >> > files out - it normally takes several minutes and has always
> > >> done.
> > >> > > >> >
> > >> > > >> > I looked at SF but only see 16275 in test (updated 3 days
> ago)
> > so
> > >> > > maybe
> > >> > > >> it
> > >> > > >> > hasn't made its way live yet.
> > >> > > >> >
> > >> > > >> > On Wed, Oct 5, 2016 at 10:33 AM, Thomas Eckardt
> > >> > > >> > <thomas.ecka...@thockar.com>
> > >> > > >> > wrote:
> > >> > > >> >
> > >> > > >> > > I've provided an updated assp.pl (2.5.4 16279) in CVS
> /test.
> > >> This
> > >> > > >> > version
> > >> > > >> > > shows some more information, if 'WorkerLog' is set to
> > >> diagnostic.
> > >> > > >> > >
> > >> > > >> > > Thomas
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > Von: cw <colin.war...@gmail.com>
> > >> > > >> > > An: ASSP development mailing list
> > >> > > >> <assp-test@lists.sourceforge.net>
> > >> > > >> > > Datum: 05.10.2016 10:10
> > >> > > >> > > Betreff: Re: [Assp-test] unable to detect any
> running
> > >> > worker
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > Hi Thomas,
> > >> > > >> > >
> > >> > > >> > > Thanks for chipping in. All modules are installed by
> running
> > >> the
> > >> > > >> latest
> > >> > > >> > > mod_inst.pl. Crypt::GOST on Ubuntu actually has a bug in
> it
> > so
> > >> it
> > >> > > >> > requires
> > >> > > >> > > a minor edit to the code to get it to build. So that
> module
> > was
> > >> > > >> > installed
> > >> > > >> > > by switching to /root/.cpan/build/Crypt-GOST-x-x-x and
> > >> running:
> > >> > > >> > > make clean
> > >> > > >> > > perl Makefile.PL
> > >> > > >> > > make
> > >> > > >> > > make test
> > >> > > >> > > make install
> > >> > > >> > >
> > >> > > >> > > I run cpan-outdated -p|cpanm from time to time in order to
> > keep
> > >> > > >> modules
> > >> > > >> > up
> > >> > > >> > > to date as well so things don't stay stuck on old
> versions.
> > >> > > >> > >
> > >> > > >> > > Everything that is installed now is a completely fresh
> build
> > as
> > >> > the
> > >> > > >> > > Upgrade
> > >> > > >> > > to 16.04 replaced perl 5.18 with 5.22 and all the modules
> > >> > therefore
> > >> > > >> had
> > >> > > >> > to
> > >> > > >> > > be installed from scratch.
> > >> > > >> > >
> > >> > > >> > > I bypassed the issue by truncating the tables and setting
> up
> > >> the
> > >> > > users
> > >> > > >> > > again. With there only being one user it was easier to do
> > that
> > >> > than
> > >> > > >> muck
> > >> > > >> > > about with it - especially with the other bigger issue at
> > hand.
> > >> > > >> > >
> > >> > > >> > > On Wed, Oct 5, 2016 at 8:59 AM, Thomas Eckardt
> > >> > > >> > > <thomas.ecka...@thockar.com>
> > >> > > >> > > wrote:
> > >> > > >> > >
> > >> > > >> > > > >to replace the corrupted encrypted
> > >> > > >> > > > strings with the correct values
> > >> > > >> > > >
> > >> > > >> > > > Was the 'Crypt::GOST' module from the SF download page
> at
> > the
> > >> > old
> > >> > > >> assp
> > >> > > >> > > > instance?
> > >> > > >> > > > If it was, did you install the 'Crypt::GOST' module from
> > the
> > >> SF
> > >> > > >> > download
> > >> > > >> > > > page, before you started the new assp instance?
> > >> > > >> > > >
> > >> > > >> > > > https://sourceforge.net/projects/assp/files/ASSP%20V2%
> > >> > > >> > > >
> > >> 20multithreading/ASSP%20V2%20module%20installation/Crypt-GOST/
> > >> > > >> > > >
> > >> > > >> > > > Thomas
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > Von: cw <colin.war...@gmail.com>
> > >> > > >> > > > An: ASSP development mailing list
> > >> > > >> > <assp-test@lists.sourceforge.net>
> > >> > > >> > > > Datum: 05.10.2016 09:52
> > >> > > >> > > > Betreff: Re: [Assp-test] unable to detect any
> > running
> > >> > > worker
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > Cheers for the reply.
> > >> > > >> > > >
> > >> > > >> > > > I don't think backups are an option seen as I've moved
> > >> > completely
> > >> > > >> from
> > >> > > >> > > > Ubuntu 14.04 to Ubuntu 16.04.
> > >> > > >> > > >
> > >> > > >> > > > Also, this issue has been around for months. It was
> > causing a
> > >> > > >> handful
> > >> > > >> > of
> > >> > > >> > > > shutdowns a week with the occasional spat of more
> frequent
> > >> > > >> shutdowns.
> > >> > > >> > It
> > >> > > >> > > > is
> > >> > > >> > > > entirely possible that the errors are behaviour related
> > and
> > >> > > nothing
> > >> > > >> to
> > >> > > >> > > do
> > >> > > >> > > > with the upgrade and that current email behaviour is
> > >> triggering
> > >> > > it
> > >> > > >> big
> > >> > > >> > > > style.
> > >> > > >> > > >
> > >> > > >> > > > I'm not convinced though, unfortunately I'm not
> convinced
> > of
> > >> > > >> anything
> > >> > > >> > > else
> > >> > > >> > > > hence not having much to go on. I can't see any
> > consistencies
> > >> > in
> > >> > > the
> > >> > > >> > > > behaviour leading up to the events.
> > >> > > >> > > >
> > >> > > >> > > > I don't think it is database related. It happened on one
> > of
> > >> the
> > >> > > mail
> > >> > > >> > > > servers during the upgrade to 16.04 when ASSP had just
> > >> started
> > >> > > but I
> > >> > > >> > had
> > >> > > >> > > > not yet got into the web interface to replace the
> > corrupted
> > >> > > >> encrypted
> > >> > > >> > > > strings with the correct values so all database
> > connections
> > >> > were
> > >> > > in
> > >> > > >> > > error.
> > >> > > >> > > >
> > >> > > >> > > > The problem has already started again this morning so I
> > can
> > >> see
> > >> > > this
> > >> > > >> > > being
> > >> > > >> > > > another fun day that either leads to a fix or having to
> > put
> > >> > > >> something
> > >> > > >> > > else
> > >> > > >> > > > in place.
> > >> > > >> > > >
> > >> > > >> > > > On Wed, Oct 5, 2016 at 4:34 AM, K Post
> > <nntp.p...@gmail.com>
> > >> > > wrote:
> > >> > > >> > > >
> > >> > > >> > > > > I've been reading here, but I haven't had anything to
> > >> > suggest.
> > >> > > All
> > >> > > >> > > > seems
> > >> > > >> > > > > quite odd if it was working prior to upgrading and
> > >> > downgrading
> > >> > > >> > didn't
> > >> > > >> > > > work.
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > > > Could you spin up a backup of the installation after
> > >> copying
> > >> > > the
> > >> > > >> > > current
> > >> > > >> > > > > data? Sure you'd have an older corpus, but I'd think
> > you
> > >> > could
> > >> > > >> add
> > >> > > >> > > the
> > >> > > >> > > > new
> > >> > > >> > > > > files if necessary, manually replace whitelist etc.
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > > > On Tue, Oct 4, 2016 at 5:48 PM, cw
> > <colin.war...@gmail.com
> > >> >
> > >> > > >> wrote:
> > >> > > >> > > > >
> > >> > > >> > > > > > Further development on this today, very little.
> > >> > > >> > > > > > I have moved both servers onto Ubuntu 16.04 LTS
> which
> > >> means
> > >> > > >> going
> > >> > > >> > > from
> > >> > > >> > > > > perl
> > >> > > >> > > > > > 5.18 to 5.22 and rebuilding all perl modules from
> > >> scratch.
> > >> > > >> > > > > >
> > >> > > >> > > > > > The admin user db did not work after the upgrade so
> I
> > had
> > >> > to
> > >> > > >> empty
> > >> > > >> > > the
> > >> > > >> > > > > > tables before it would come back online.
> > >> > > >> > > > > >
> > >> > > >> > > > > > I'm still getting delayed emails and assp shutting
> > down
> > >> > > telling
> > >> > > >> me
> > >> > > >> > > it
> > >> > > >> > > > is
> > >> > > >> > > > > > unable
> > >> > > >> > > > > > to detect any running worker.
> > >> > > >> > > > > >
> > >> > > >> > > > > > If this goes on much longer the MD will pull the
> plug
> > and
> > >> > > we'll
> > >> > > >> > end
> > >> > > >> > > up
> > >> > > >> > > > > > moving to a third party solution which is not
> > something I
> > >> > > want
> > >> > > >> but
> > >> > > >> > > if
> > >> > > >> > > > I
> > >> > > >> > > > > > can't fix it I can't defend it :/
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > ------------------------------------------------------------
> > >> > > >> > > > > > ------------------
> > >> > > >> > > > > > Check out the vibrant tech community on one of the
> > >> world's
> > >> > > most
> > >> > > >> > > > > > engaging tech sites, SlashDot.org!
> > >> http://sdm.link/slashdot
> > >> > > >> > > > > > _______________________________________________
> > >> > > >> > > > > > Assp-test mailing list
> > >> > > >> > > > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > > >
> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > > > >
> > >> > > >> > > > > >
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> ------------------------------------------------------------
> > >> > > >> > > > > ------------------
> > >> > > >> > > > > Check out the vibrant tech community on one of the
> > world's
> > >> > most
> > >> > > >> > > > > engaging tech sites, SlashDot.org!
> > >> http://sdm.link/slashdot
> > >> > > >> > > > > _______________________________________________
> > >> > > >> > > > > Assp-test mailing list
> > >> > > >> > > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > > >
> > >> > > >> > > > >
> > >> > > >> > > >
> > ------------------------------------------------------------
> > >> > > >> > > > ------------------
> > >> > > >> > > > Check out the vibrant tech community on one of the
> world's
> > >> most
> > >> > > >> > > > engaging tech sites, SlashDot.org!
> > http://sdm.link/slashdot
> > >> > > >> > > > _______________________________________________
> > >> > > >> > > > Assp-test mailing list
> > >> > > >> > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > > DISCLAIMER:
> > >> > > >> > > > *******************************************************
> > >> > > >> > > > This email and any files transmitted with it may be
> > >> > confidential,
> > >> > > >> > > legally
> > >> > > >> > > > privileged and protected in law and are intended solely
> > for
> > >> the
> > >> > > use
> > >> > > >> of
> > >> > > >> > > the
> > >> > > >> > > >
> > >> > > >> > > > individual to whom it is addressed.
> > >> > > >> > > > This email was multiple times scanned for viruses. There
> > >> should
> > >> > > be
> > >> > > >> no
> > >> > > >> > > > known virus in this email!
> > >> > > >> > > > *******************************************************
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > > >
> > ------------------------------------------------------------
> > >> > > >> > > > ------------------
> > >> > > >> > > > Check out the vibrant tech community on one of the
> world's
> > >> most
> > >> > > >> > > > engaging tech sites, SlashDot.org!
> > http://sdm.link/slashdot
> > >> > > >> > > > _______________________________________________
> > >> > > >> > > > Assp-test mailing list
> > >> > > >> > > > Assp-test@lists.sourceforge.net
> > >> > > >> > > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > > >
> > >> > > >> > > >
> > >> > > >> > >
> ------------------------------------------------------------
> > >> > > >> > > ------------------
> > >> > > >> > > Check out the vibrant tech community on one of the world's
> > most
> > >> > > >> > > engaging tech sites, SlashDot.org!
> http://sdm.link/slashdot
> > >> > > >> > > _______________________________________________
> > >> > > >> > > Assp-test mailing list
> > >> > > >> > > Assp-test@lists.sourceforge.net
> > >> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > > DISCLAIMER:
> > >> > > >> > > *******************************************************
> > >> > > >> > > This email and any files transmitted with it may be
> > >> confidential,
> > >> > > >> > legally
> > >> > > >> > > privileged and protected in law and are intended solely
> for
> > the
> > >> > use
> > >> > > of
> > >> > > >> > the
> > >> > > >> > >
> > >> > > >> > > individual to whom it is addressed.
> > >> > > >> > > This email was multiple times scanned for viruses. There
> > should
> > >> > be
> > >> > > no
> > >> > > >> > > known virus in this email!
> > >> > > >> > > *******************************************************
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > >
> ------------------------------------------------------------
> > >> > > >> > > ------------------
> > >> > > >> > > Check out the vibrant tech community on one of the world's
> > most
> > >> > > >> > > engaging tech sites, SlashDot.org!
> http://sdm.link/slashdot
> > >> > > >> > > _______________________________________________
> > >> > > >> > > Assp-test mailing list
> > >> > > >> > > Assp-test@lists.sourceforge.net
> > >> > > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> > >
> > >> > > >> > >
> > >> > > >> > ------------------------------------------------------------
> > >> > > >> > ------------------
> > >> > > >> > Check out the vibrant tech community on one of the world's
> > most
> > >> > > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > >> > _______________________________________________
> > >> > > >> > Assp-test mailing list
> > >> > > >> > Assp-test@lists.sourceforge.net
> > >> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > DISCLAIMER:
> > >> > > >> > *******************************************************
> > >> > > >> > This email and any files transmitted with it may be
> > confidential,
> > >> > > >> legally
> > >> > > >> > privileged and protected in law and are intended solely for
> > the
> > >> use
> > >> > > of
> > >> > > >> the
> > >> > > >> >
> > >> > > >> > individual to whom it is addressed.
> > >> > > >> > This email was multiple times scanned for viruses. There
> > should
> > >> be
> > >> > no
> > >> > > >> > known virus in this email!
> > >> > > >> > *******************************************************
> > >> > > >> >
> > >> > > >> >
> > >> > > >> > ------------------------------------------------------------
> > >> > > >> > ------------------
> > >> > > >> > Check out the vibrant tech community on one of the world's
> > most
> > >> > > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > >> > _______________________________________________
> > >> > > >> > Assp-test mailing list
> > >> > > >> > Assp-test@lists.sourceforge.net
> > >> > > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >> >
> > >> > > >> >
> > >> > > >> ------------------------------------------------------------
> > >> > > >> ------------------
> > >> > > >> Check out the vibrant tech community on one of the world's
> most
> > >> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > >> _______________________________________________
> > >> > > >> Assp-test mailing list
> > >> > > >> Assp-test@lists.sourceforge.net
> > >> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >> DISCLAIMER:
> > >> > > >> *******************************************************
> > >> > > >> This email and any files transmitted with it may be
> > confidential,
> > >> > > legally
> > >> > > >> privileged and protected in law and are intended solely for
> the
> > use
> > >> > of
> > >> > > the
> > >> > > >>
> > >> > > >> individual to whom it is addressed.
> > >> > > >> This email was multiple times scanned for viruses. There
> should
> > be
> > >> no
> > >> > > >> known virus in this email!
> > >> > > >> *******************************************************
> > >> > > >>
> > >> > > >>
> > >> > > >> ------------------------------------------------------------
> > >> > > >> ------------------
> > >> > > >> Check out the vibrant tech community on one of the world's
> most
> > >> > > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > >> _______________________________________________
> > >> > > >> Assp-test mailing list
> > >> > > >> Assp-test@lists.sourceforge.net
> > >> > > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > > >>
> > >> > > >>
> > >> > > >
> > >> > > ------------------------------------------------------------
> > >> > > ------------------
> > >> > > Check out the vibrant tech community on one of the world's most
> > >> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > _______________________________________________
> > >> > > Assp-test mailing list
> > >> > > Assp-test@lists.sourceforge.net
> > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > DISCLAIMER:
> > >> > > *******************************************************
> > >> > > This email and any files transmitted with it may be confidential,
> > >> > legally
> > >> > > privileged and protected in law and are intended solely for the
> use
> > of
> > >> > the
> > >> > >
> > >> > > individual to whom it is addressed.
> > >> > > This email was multiple times scanned for viruses. There should
> be
> > no
> > >> > > known virus in this email!
> > >> > > *******************************************************
> > >> > >
> > >> > >
> > >> > > ------------------------------------------------------------
> > >> > > ------------------
> > >> > > Check out the vibrant tech community on one of the world's most
> > >> > > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > > _______________________________________________
> > >> > > Assp-test mailing list
> > >> > > Assp-test@lists.sourceforge.net
> > >> > > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> > >
> > >> > >
> > >> > ------------------------------------------------------------
> > >> > ------------------
> > >> > Check out the vibrant tech community on one of the world's most
> > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > _______________________________________________
> > >> > Assp-test mailing list
> > >> > Assp-test@lists.sourceforge.net
> > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > DISCLAIMER:
> > >> > *******************************************************
> > >> > This email and any files transmitted with it may be confidential,
> > >> legally
> > >> > privileged and protected in law and are intended solely for the use
> > of
> > >> the
> > >> >
> > >> > individual to whom it is addressed.
> > >> > This email was multiple times scanned for viruses. There should be
> no
> > >> > known virus in this email!
> > >> > *******************************************************
> > >> >
> > >> >
> > >> > ------------------------------------------------------------
> > >> > ------------------
> > >> > Check out the vibrant tech community on one of the world's most
> > >> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> > _______________________________________________
> > >> > Assp-test mailing list
> > >> > Assp-test@lists.sourceforge.net
> > >> > https://lists.sourceforge.net/lists/listinfo/assp-test
> > >> >
> > >> >
> > >> ------------------------------------------------------------
> > >> ------------------
> > >> Check out the vibrant tech community on one of the world's most
> > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> _______________________________________________
> > >> Assp-test mailing list
> > >> Assp-test@lists.sourceforge.net
> > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >>
> > >>
> > >>
> > >>
> > >> DISCLAIMER:
> > >> *******************************************************
> > >> This email and any files transmitted with it may be confidential,
> > legally
> > >> privileged and protected in law and are intended solely for the use
> of
> > the
> > >>
> > >> individual to whom it is addressed.
> > >> This email was multiple times scanned for viruses. There should be no
> > >> known virus in this email!
> > >> *******************************************************
> > >>
> > >>
> > >> ------------------------------------------------------------
> > >> ------------------
> > >> Check out the vibrant tech community on one of the world's most
> > >> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > >> _______________________________________________
> > >> Assp-test mailing list
> > >> Assp-test@lists.sourceforge.net
> > >> https://lists.sourceforge.net/lists/listinfo/assp-test
> > >>
> > >>
> > >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> >
> >
> > DISCLAIMER:
> > *******************************************************
> > This email and any files transmitted with it may be confidential,
> legally
> > privileged and protected in law and are intended solely for the use of
> the
> >
> > individual to whom it is addressed.
> > This email was multiple times scanned for viruses. There should be no
> > known virus in this email!
> > *******************************************************
> >
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Assp-test mailing list
> > Assp-test@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/assp-test
> >
> >
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
>
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
