>2016-10-18 15:19 GMT+02:00 phrogz <phr...@phrogz.me>:
>>
>>
>>
>>>De : rsyslog-boun...@lists.adiscon.com <rsyslog-boun...@lists.adiscon.com> 
>>>de la part de Rainer Gerhards <rgerha...@hq.adiscon.com>
>>>Envoyé : mardi 18 octobre 2016 12:48
>>>À : rsyslog-users
>>>Objet : Re: [rsyslog] Action Queue misunderstanding
>>>
>>>2016-10-18 12:26 GMT+02:00 phrogz <phr...@phrogz.me>:
>>>> Hello,
>>>>
>>>>
>>>> 1/ I'm trying to understand why in impstats my action queue doesn't reach 
>>>> an higher value than 1000 (and sometime there is a value for 'full', I 
>>>> don't know why).
>>>
>>>you have commented out "queue.size", so it uses the default of 1,000.
>>>If you specify a larger size, that larger value will be used.
>>>
>>>This is the same for DA as for other queues. What did you get the idea
>>>it does not apply to DA?
>>>
>>>Rainer
>>
>>
>> Thanks, that solved the issue in the point 1!
>> I think I misundertood the following:
>> Here: http://www.rsyslog.com/doc/v8-stable/concepts/queues.html ; Chapter 
>> "Limiting the Queue Size": "[...]Disk assisted queues are special in that 
>> they do not have any size limit.[...]"
>>
>> I still have the issue described in the second point.
>
>I overlooked this. If you do not have at least 8.21.0, update to
>current 8.22.0. It is far more robust in regard to queue failures.
>
>Rainer
>>
>> Thanks

I've the version 8.21, I will try an update to 8.22

Ludovic

>>
>>
>>>>
>>>>
>>>> e.g. :
>>>>
>>>>       "name": "main Q",
>>>>       "origin": "core.queue",
>>>>       "size": 10916,
>>>>       "enqueued": 23215,
>>>>       "full": 0,
>>>>       "discarded_full": 0,
>>>>       "discarded_nf": 0,
>>>>       "maxqsize": 144059
>>>>
>>>>       "name": "action 1 queue",
>>>>       "origin": "core.queue",
>>>>       "size": 1000,
>>>>       "enqueued": 17808,
>>>>       "full": 63,
>>>>       "discarded_full": 0,
>>>>       "discarded_nf": 0,
>>>>       "maxqsize": 1000
>>>>
>>>>       "name": "action 2 queue",
>>>>       "origin": "core.queue",
>>>>       "size": 0,
>>>>       "enqueued": 0,
>>>>       "full": 0,
>>>>       "discarded_full": 0,
>>>>       "discarded_nf": 0,
>>>>       "maxqsize": 1000
>>>>
>>>>       "name": "action 1 queue[DA]",
>>>>       "origin": "core.queue",
>>>>       "size": 0,
>>>>       "enqueued": 0,
>>>>       "full": 0,
>>>>       "discarded_full": 0,
>>>>       "discarded_nf": 0,
>>>>       "maxqsize": 0
>>>>
>>>>       "name": "action 2 queue[DA]",
>>>>       "origin": "core.queue",
>>>>       "size": 0,
>>>>       "enqueued": 0,
>>>>       "full": 0,
>>>>       "discarded_full": 0,
>>>>       "discarded_nf": 0,
>>>>       "maxqsize": 0
>>>>
>>>>
>>>> 2/ There is also something strange, there is two old queues in the rsyslog 
>>>> spool that are not deleted.
>>>> #####################
>>>> #####################
>>>> #####################
>>>>
>>>> here is the debug:
>>>> 1768.304204149:main thread    : action 1 queue: starting queue
>>>> 1768.304209377:main thread    : action 1 queue: is disk-assisted, disk 
>>>> will be used on demand
>>>> 1768.304216886:main thread    : action 1 queue: params: type 1, enq-only 
>>>> 0, disk assisted 1, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, 
>>>> maxQSize 1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 
>>>> 700, deq batch size 1000, high wtrmrk 900, low wtrmrk 700, discardmrk 980, 
>>>> max wrkr 4, min msgs f. wrkr 250
>>>> 1768.304222849:main thread    : action 1 queue:Reg: finalizing 
>>>> construction of worker thread pool (numworkerThreads 4)
>>>> 1768.304228002:main thread    : action 1 queue:Reg/w0: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304236533:main thread    : action 1 queue:Reg/w1: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304243582:main thread    : action 1 queue:Reg/w2: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304260052:main thread    : action 1 queue:Reg/w3: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304275153:main thread    : action 1 queue:DAwpool: finalizing 
>>>> construction of worker thread pool (numworkerThreads 1)
>>>> 1768.304288963:main thread    : action 1 queue:DAwpool/w0: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304334755:main thread    : action 1 queue[DA]: starting queue
>>>> 1768.304343275:main thread    : action 1 queue[DA]: .qi file name is 
>>>> '/apps/rsyslog/spool/ToLogStash.qi', len 33
>>>> 1768.304347249:main thread    : action 1 queue[DA]: I am a child
>>>> 1768.304519153:main thread    : file stream 
>>>> /apps/rsyslog/spool/ToLogStash.qi params: flush interval 0, async write 0
>>>> 1768.304545844:main thread    : file '/apps/rsyslog/spool/ToLogStash.qi' 
>>>> opened as #10 with mode 384
>>>> 1768.304565987:main thread    : strm 0x7f43404b9890: opened file 
>>>> '/apps/rsyslog/spool/ToLogStash.qi' for READ as 10
>>>> 1768.304574600:main thread    : strm 0x7f43404b9890: file 10 read 543 bytes
>>>> 1768.304603730:main thread    : file stream ToLogStash params: flush 
>>>> interval 0, async write 0
>>>> 1768.304626194:main thread    : file stream ToLogStash params: flush 
>>>> interval 0, async write 0
>>>> 1768.304636987:main thread    : file stream ToLogStash params: flush 
>>>> interval 0, async write 0
>>>> 1768.304658150:main thread    : file 
>>>> '/apps/rsyslog/spool/ToLogStash.00038759' opened as #11 with mode 384
>>>> 1768.304669771:main thread    : queue 'strm 0x7f43404bac20', file 
>>>> '/apps/rsyslog/spool/ToLogStash.00038759' opened for non-append write, but 
>>>> already contains 358469 bytes
>>>> 1768.304674986:main thread    : strm 0x7f43404bac20: opened file 
>>>> '/apps/rsyslog/spool/ToLogStash.00038759' for WRITE as 11
>>>> 1768.304679648:main thread    : strm 0x7f43404bac20: file 11 seek, pos 639
>>>> 1768.304691562:main thread    : file 
>>>> '/apps/rsyslog/spool/ToLogStash.00038759' opened as #22 with mode 384
>>>> 1768.304701188:main thread    : strm 0x7f43404bbeb0: opened file 
>>>> '/apps/rsyslog/spool/ToLogStash.00038759' for READ as 22
>>>> 1768.304705945:main thread    : strm 0x7f43404bbeb0: file 22 seek, pos 639
>>>> 1768.304716357:main thread    : file 
>>>> '/apps/rsyslog/spool/ToLogStash.00038759' opened as #23 with mode 384
>>>> 1768.304726030:main thread    : strm 0x7f43404bd2a0: opened file 
>>>> '/apps/rsyslog/spool/ToLogStash.00038759' for READ as 23
>>>> 1768.304731297:main thread    : strm 0x7f43404bd2a0: file 23 seek, pos 639
>>>> 1768.304736880:main thread    : strm 0x7f43404b9890: file 
>>>> 10(/apps/rsyslog/spool/ToLogStash.qi) closing, bDeleteOnClose 0
>>>> 1768.304750072:main thread    : action 1 queue[DA]: params: type 2, 
>>>> enq-only 0, disk assisted 0, spoolDir '/apps/rsyslog/spool', maxFileSz 
>>>> 1048576, maxQSize 0, lqsize 0, pqsize 0, child 1, full delay -1, light 
>>>> delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max 
>>>> wrkr 1, min msgs f. wrkr 0
>>>> 1768.304758727:main thread    : action 1 queue[DA]:Reg: finalizing 
>>>> construction of worker thread pool (numworkerThreads 1)
>>>> 1768.304764348:main thread    : action 1 queue[DA]:Reg/w0: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304769386:main thread    : action 1 queue[DA]: queue finished 
>>>> initialization
>>>> 1768.304776843:main thread    : action 1 queue: DA queue initialized, disk 
>>>> queue 0x7f43404b9550
>>>> 1768.304781135:main thread    : action 1 queue: queue finished 
>>>> initialization
>>>> 1768.304789476:main thread    : Action builtin:omfwd[0x7f4340483dc0]: 
>>>> queue 0x7f4340484b60 started
>>>> 1768.304794344:main thread    : iterateAllActions calling into action 
>>>> 0x7f4340484ea0
>>>> 1768.304798231:main thread    : action 2 queue: starting queue
>>>> 1768.304803919:main thread    : action 2 queue: is disk-assisted, disk 
>>>> will be used on demand
>>>> 1768.304811645:main thread    : action 2 queue: params: type 1, enq-only 
>>>> 0, disk assisted 1, spoolDir '/apps/rsyslog/spool', maxFileSz 1048576, 
>>>> maxQSize 1000, lqsize 0, pqsize 0, child 0, full delay 450, light delay 
>>>> 700, deq batch size 1000, high wtrmrk 900, low wtrmrk 700, discardmrk 980, 
>>>> max wrkr 4, min msgs f. wrkr 250
>>>> 1768.304817986:main thread    : action 2 queue:Reg: finalizing 
>>>> construction of worker thread pool (numworkerThreads 4)
>>>> 1768.304823043:main thread    : action 2 queue:Reg/w0: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304840673:main thread    : action 2 queue:Reg/w1: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304862659:main thread    : action 2 queue:Reg/w2: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304879227:main thread    : action 2 queue:Reg/w3: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304894375:main thread    : action 2 queue:DAwpool: finalizing 
>>>> construction of worker thread pool (numworkerThreads 1)
>>>> 1768.304899909:main thread    : action 2 queue:DAwpool/w0: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.304916529:main thread    : action 2 queue[DA]: starting queue
>>>> 1768.304922466:main thread    : action 2 queue[DA]: .qi file name is 
>>>> '/apps/rsyslog/spool/RelayToLogStash.qi', len 38
>>>> 1768.304926530:main thread    : action 2 queue[DA]: I am a child
>>>> 1768.304938902:main thread    : file stream 
>>>> /apps/rsyslog/spool/RelayToLogStash.qi params: flush interval 0, async 
>>>> write 0
>>>> 1768.304955158:main thread    : file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.qi' opened as #10 with mode 384
>>>> 1768.304965285:main thread    : strm 0x7f43404cb7c0: opened file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.qi' for READ as 10
>>>> 1768.304985578:main thread    : strm 0x7f43404cb7c0: file 10 read 560 bytes
>>>> 1768.305009054:main thread    : file stream RelayToLogStash params: flush 
>>>> interval 0, async write 0
>>>> 1768.305064267:main thread    : file stream RelayToLogStash params: flush 
>>>> interval 0, async write 0
>>>> 1768.305074762:main thread    : file stream RelayToLogStash params: flush 
>>>> interval 0, async write 0
>>>> 1768.305151652:main thread    : file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #24 with mode 384
>>>> 1768.305165535:main thread    : queue 'strm 0x7f43404ccbe0', file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened for non-append 
>>>> write, but already contains 1012893 bytes
>>>> 1768.305170609:main thread    : strm 0x7f43404ccbe0: opened file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.00002857' for WRITE as 24
>>>> 1768.305175510:main thread    : strm 0x7f43404ccbe0: file 24 seek, pos 
>>>> 438297
>>>> 1768.305187379:main thread    : file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #25 with mode 384
>>>> 1768.305197024:main thread    : strm 0x7f43404cdfd0: opened file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.00002857' for READ as 25
>>>> 1768.305202122:main thread    : strm 0x7f43404cdfd0: file 25 seek, pos 1817
>>>> 1768.305212704:main thread    : file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.00002857' opened as #26 with mode 384
>>>> 1768.305221830:main thread    : strm 0x7f43404cf260: opened file 
>>>> '/apps/rsyslog/spool/RelayToLogStash.00002857' for READ as 26
>>>> 1768.305226641:main thread    : strm 0x7f43404cf260: file 26 seek, pos 1817
>>>> 1768.305232314:main thread    : strm 0x7f43404cb7c0: file 
>>>> 10(/apps/rsyslog/spool/RelayToLogStash.qi) closing, bDeleteOnClose 0
>>>> 1768.305244857:main thread    : action 2 queue[DA]: params: type 2, 
>>>> enq-only 0, disk assisted 0, spoolDir '/apps/rsyslog/spool', maxFileSz 
>>>> 1048576, maxQSize 0, lqsize 731, pqsize 731, child 1, full delay -1, light 
>>>> delay -1, deq batch size 8, high wtrmrk 0, low wtrmrk 1, discardmrk 0, max 
>>>> wrkr 1, min msgs f. wrkr 0
>>>> 1768.305253009:main thread    : action 2 queue[DA]:Reg: finalizing 
>>>> construction of worker thread pool (numworkerThreads 1)
>>>> 1768.305258858:main thread    : action 2 queue[DA]:Reg/w0: finalizing 
>>>> construction of worker instance data (for 13 actions)
>>>> 1768.305263853:main thread    : action 2 queue[DA]: queue finished 
>>>> initialization
>>>> 1768.305268754:main thread    : action 2 queue[DA]:Reg: high activity - 
>>>> starting 1 additional worker thread(s).
>>>> 1768.305374179:main thread    : action 2 queue[DA]:Reg: started with state 
>>>> 0, num workers now 1
>>>> 1768.305385824:main thread    : action 2 queue: DA queue initialized, disk 
>>>> queue 0x7f43404cb3f0
>>>> 1768.305390224:main thread    : action 2 queue: queue finished 
>>>> initialization
>>>> 1768.305397556:main thread    : Action builtin:omfwd[0x7f4340484ea0]: 
>>>> queue 0x7f4340485460 started
>>>> 1768.305402568:main thread    : Activating Ruleset Queue[(nil)] for 
>>>> Ruleset RSYSLOG_DefaultRuleset
>>>> 1768.305406401:main thread    : Activating Ruleset Queue[(nil)] for 
>>>> Ruleset local
>>>> 1768.305417331:7f433317b700: thread created, tid 4945, name 'rs:action 2 
>>>> queue[DA]:R'
>>>> 1768.305498188:action 2 queue[DA]:Reg/w0: wti 0x7f43404cb8f0: worker 
>>>> starting
>>>> 1768.305512959:main thread    : Activating Ruleset Queue[(nil)] for 
>>>> Ruleset remote_tcp_secure_6514
>>>> 1768.305536551:main thread    : Activating Ruleset Queue[(nil)] for 
>>>> Ruleset remote_relay_secure_6514
>>>> 1768.305542715:main thread    : activateMainQueue: mainq cnf obj ptr is 
>>>> 0x7f434047ebb0
>>>> 1768.305554811:main thread    : nvlstGetParam: name 'queue.size', type 9, 
>>>> valnode->bUsed 0
>>>> 1768.305560230:main thread    : nvlstGetParam: name 
>>>> 'queue.dequeuebatchsize', type 6, valnode->bUsed 0
>>>> 1768.305567499:main thread    : nvlstGetParam: name 'queue.workerthreads', 
>>>> type 6, valnode->bUsed 0
>>>> 1768.305573965:main thread    : queue param blk:
>>>> 1768.305577887:main thread    : queue.filename: (unset)
>>>> 1768.305594020:action 2 queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted 
>>>> 0 objects and enqueued 0 objects
>>>> 1768.305616369:action 2 queue[DA]:Reg/w0: doDeleteBatch: delete batch from 
>>>> store, new sizes: log 731, phys 731
>>>> 1768.305630344:main thread    : queue.spooldirectory: (unset)
>>>> 1768.305640736:action 2 queue[DA]:Reg/w0: strm 0x7f43404cf260: file 26 
>>>> read 4096 bytes
>>>>
>>>> 3/Here is my config:
>>>> ############################
>>>> ############################
>>>> ############################
>>>> global(
>>>>   defaultNetstreamDriver="gtls"
>>>>   #debug.gnutls="10"
>>>>   #TCP TLS
>>>>   defaultNetstreamDriverCAFile="/apps/rsyslog/certCA.crt"
>>>>   defaultNetstreamDriverCertFile="/apps/rsyslog/certHost.crt"
>>>>   defaultNetstreamDriverKeyFile="/apps/rsyslog/certHost.key"
>>>> )
>>>> module(load="omrelp")
>>>> module(load="imptcp")
>>>> module(
>>>>   load="imtcp"
>>>>   streamDriver.name="gtls"
>>>>   streamDriver.mode="1"
>>>>   streamDriver.authMode="x509/name"
>>>>   permittedPeer=["sourcesrv1.test.lan","sourcesrv2.test.lan"]
>>>> )
>>>> module(
>>>>  load="impstats"
>>>>  interval="10"
>>>>  resetCounters="on"
>>>>  format="cee"
>>>>  ruleset="remote_tcp_secure_6514"      # send stats to the 
>>>> remote_tcp_secure_6514 ruleset
>>>>  )
>>>>
>>>>
>>>> template(name="SyslogSource" type="string"
>>>>          string="<%pri%>%timestamp:::date-rfc3339% %hostname% 
>>>> %fromhost-ip% %syslogtag% %msg%\n"
>>>>         )
>>>> template(name="SyslogSourceRelay" type="string"
>>>>          string="<%pri%>%timestamp:::date-rfc3339% %hostname% %syslogtag% 
>>>> %msg%\n"
>>>>         )
>>>>
>>>> /* Input entries */
>>>>
>>>> input(type="imptcp" port="514" ruleset="remote_tcp_secure_6514")
>>>> #input(type="imptcp" port="514" ruleset="remote_relp_secure_6514")
>>>>
>>>> input(
>>>>   type="imtcp"
>>>>   port="6514"
>>>>   name="imtcp_secure"
>>>>   ruleset="remote_relay_secure_6514"
>>>> )
>>>>
>>>> main_queue(
>>>>   queue.size="1000000"   # capacity of the main queue
>>>>   queue.dequeuebatchsize="10000"  # process messages in batches of 1000 
>>>> and move them to the action queues
>>>>   queue.workerthreads="2"  # 2 threads for the main queue
>>>> )
>>>>
>>>> /* Remote Logging */
>>>>
>>>> RuleSet (name="remote_tcp_secure_6514")
>>>> {
>>>>     action(
>>>>       type="omfwd"
>>>>       target="destserv.test.lan"
>>>>       port="6514"
>>>>       protocol="tcp"
>>>>       StreamDriverMode="1"
>>>>       StreamDriver="gtls"
>>>>       StreamDriverAuthMode="x509/name"
>>>>       StreamDriverPermittedPeers="destserv.test.lan"
>>>>       action.resumeRetryCount="-1"
>>>>       queue.type="linkedList"
>>>>       queue.filename="ToLogStash"
>>>>       queue.spoolDirectory="/apps/rsyslog/spool"
>>>>       queue.maxDiskSpace="298g"
>>>>       queue.saveOnShutdown="on"
>>>>       queue.dequeuebatchsize="2000"
>>>>       RebindInterval="1000"
>>>>       queue.workerthreads="4"
>>>>       queue.highwatermark="500000"
>>>>       queue.lowwatermark="200000"
>>>>       template="SyslogSource"
>>>>       #queue.size="1000000000" #Not needed for DA Queue
>>>>       action.reportSuspension="on"
>>>>       action.reportSuspensionContinuation="on"
>>>>     )
>>>> }
>>>>
>>>>
>>>> RuleSet (name="remote_relay_secure_6514")
>>>> {
>>>>     action(
>>>>       type="omfwd"
>>>>       target="destserv.test.lan"
>>>>       port="6514"
>>>>       protocol="tcp"
>>>>       StreamDriverMode="1"
>>>>       StreamDriver="gtls"
>>>>       StreamDriverAuthMode="x509/name"
>>>>       StreamDriverPermittedPeers="destserv.test.lan"
>>>>       action.resumeRetryCount="-1"
>>>>       queue.type="linkedList"
>>>>       queue.filename="RelayToLogStash"
>>>>       queue.spoolDirectory="/apps/rsyslog/spool"
>>>>       queue.maxDiskSpace="200g"
>>>>       queue.saveOnShutdown="on"
>>>>       queue.dequeuebatchsize="2000"
>>>>       RebindInterval="1000"
>>>>       queue.workerthreads="4"
>>>>       queue.highwatermark="500000"
>>>>       queue.lowwatermark="200000"
>>>>       template="SyslogSourceRelay"
>>>>       #queue.size="1000000000" #Not needed for DA Queue
>>>>       action.reportSuspension="on"
>>>>       action.reportSuspensionContinuation="on"
>>>>     )
>>>> }
>>>>
>>>> Thanks,
>>>>
>>>> Ludovic
>>>>
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad 
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you 
>>>> DON'T LIKE THAT.
>>>_______________________________________________
>>>rsyslog mailing list
>>>http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>http://www.rsyslog.com/professional-services/
>>>What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>>>sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>>>LIKE THAT.
>>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
>_______________________________________________
>rsyslog mailing list
>http://lists.adiscon.net/mailman/listinfo/rsyslog
>http://www.rsyslog.com/professional-services/
>What's up with rsyslog? Follow https://twitter.com/rgerhards
>NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
>THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to