Ok.. But still am looking myself for the possibility to integrate with
mcollective application itself. Otherwise I need to have two separate
execution methods to be called in our automation which is very difficult to
handle it.
Between I could see the batch mode works fine with rpc but when I tries to
return the request id its throwing error, but the same works fine when I
run with -I <servername>
# mco rpc --agent shell --action execute --argument cmd="whoami"
--nodes=file --reply-to=/queue/data.consumer.all --batch 100 -v
Discovering hosts using the flatfile method .... 59
* [ ============================================================> ] 59 / 59
The rpc application failed to run: can't convert Array into String
can't convert Array into String (TypeError)
from /usr/libexec/mcollective/mcollective/application/rpc.rb:91:in
`+' <----
from /usr/libexec/mcollective/mcollective/application/rpc.rb:91:in
`main'
from /usr/lib/ruby/site_ruby/1.8/mcollective/application.rb:291:in
`run'
from /usr/lib/ruby/site_ruby/1.8/mcollective/applications.rb:23:in
`run'
from /usr/bin/mco:24
On Sunday, January 28, 2018 at 10:40:54 PM UTC+5:30, R.I.Pienaar wrote:
>
>
> On 28 Jan 2018, at 17:54, kk21987 <kkarth...@gmail.com <javascript:>>
> wrote:
>
> Great and thanks! Let me try this one, between in which location I should
> place this file to make normal mco query work with batch.
>
>
> You have to adopt this to your needs and use it instead of the normal mco
> commands for this purpose.
>
> It’s a custom behaviour basically
>
>
>
>
> On Sunday, January 28, 2018 at 9:49:53 PM UTC+5:30, R.I.Pienaar wrote:
>
>> here's a batch fire and forget client, of course it doesnt return
>> instantly but it doesnt expect replies or wait for replies
>>
>> https://gist.github.com/ripienaar/52e73db531cc22b434b8433ababd8df2
>>
>> On Sun, 28 Jan 2018, at 16:51, kk21987 wrote:
>> > It would be good if I get true batch mode where all requests have the
>> same
>> > ID, since my consumer segregates the response output based on the ID
>> only.
>> >
>> > The reason why I want to go with batch mode is, I have some set of
>> servers
>> > where I need to collect some data. But in this the data and also the
>> > servers list will keep vary which comes from different subcollectives.
>> So I
>> > don't have specific facts filter to run in broadcast mode and also I
>> can't
>> > make such facts also. Its basically kind of requirement for us to run
>> this
>> > often.
>> >
>> >
>> > On Sunday, January 28, 2018 at 9:14:52 PM UTC+5:30, R.I.Pienaar wrote:
>> >
>> > >
>> > >
>> > > On Sun, 28 Jan 2018, at 15:29, kk21987 wrote:
>> > > > Yes please. If the reply-to works well with batch mode it would be
>> great
>> > > > for me as it will solve my proble :)
>> > >
>> > > yeah fire and forget doesnt support batch mode in the normal client
>> > >
>> > > in your use case does it matter if the replies all have the same
>> request
>> > > id - ie. do you want a true batch mode where all requests have the
>> same ID
>> > > or many seperate requests would be fine?
>> > >
>> > > I still am curious why you need to use direct mode rather than
>> broadcast
>> > > >
>> > > > On Sunday, January 28, 2018 at 6:23:35 PM UTC+5:30, R.I.Pienaar
>> wrote:
>> > > > >
>> > > > >
>> > > > > On 28 Jan 2018, at 13:31, kk21987 <kkarth...@gmail.com
>> <javascript:>>
>> > > > > wrote:
>> > > > >
>> > > > > Yeah I agree it doesn't make sense to run without batch in direct
>> > > address
>> > > > > mode. But there is one problem using batch with reply-to.
>> > > > >
>> > > > > I have set the export for MCOLLECTIVE_EXTRA_OPTS with the values
>> batch
>> > > > > size as 200 and batch sleep as 30 and when I run mco query with
>> > > --reply-to
>> > > > > its returning the result on the console of mco client server
>> itself
>> > > which I
>> > > > > don't want. Since I have separate consumer server configured for
>> this.
>> > > > >
>> > > > > I really don't understand why it behaves like this. Any idea
>> please.
>> > > > >
>> > > > >
>> > > > > Could be batch mode just isn’t supported in the library when
>> using
>> > > reply
>> > > > > to. I can look later when at a computer again
>> > > > >
>> > > > > Would be trivial to write a small custom application that does
>> this
>> > > > > though.
>> > > > >
>> > > > >
>> > > > > On Sunday, January 28, 2018 at 3:46:04 PM UTC+5:30, R.I.Pienaar
>> wrote:
>> > > > >
>> > > > >>
>> > > > >>
>> > > > >> On 28 Jan 2018, at 11:13, kk21987 <kkarth...@gmail.com> wrote:
>> > > > >>
>> > > > >> Understood and thanks for the detailed clarification! I have
>> already
>> > > > >> splitted as different sub collectives (2000 per subcollective)
>> but
>> > > again
>> > > > >> all the subcollectives are connected with Broker-A which will
>> act as
>> > > > >> Central(just for naming convention) and this central will get
>> big
>> > > pressure
>> > > > >> when I run direct addressing against huge number of servers. So
>> is
>> > > there
>> > > > >> any feasibility to connect all subcollectives ActiveMQ brokers
>> and
>> > > run the
>> > > > >> command instead of using Central one by declaring some parameter
>> in
>> > > > >> mcollective client config or using any other tool in between? I
>> think
>> > > which
>> > > > >> this way it will go smooth, since I see mcollective publish the
>> > > message
>> > > > >> very quickly to Broker-A(central) and as you said this is
>> getting big
>> > > > >> pressure.
>> > > > >>
>> > > > >>
>> > > > >> Why do you need to use direct addressing for your use case? If
>> your
>> > > node
>> > > > >> lists arbitrary and not something you can discover against?
>> > > > >>
>> > > > >> You don’t use any of the confirmation features it brings?
>> > > > >>
>> > > > >> Only other option at this scale might be to direct address +
>> batches
>> > > > >>
>> > > > >>
>> > > > >> On Sunday, January 28, 2018 at 3:35:01 PM UTC+5:30, R.I.Pienaar
>> > > wrote:
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> On 28 Jan 2018, at 10:48, kk21987 <kkarth...@gmail.com> wrote:
>> > > > >>>
>> > > > >>> I agree. I read about the Choria and very interested to use it.
>> But
>> > > am
>> > > > >>> running little old version of Ruby, Puppet where I need to plan
>> it
>> > > for
>> > > > >>> upgrade. But still Choria is in active dev am little concern to
>> go
>> > > ahead it
>> > > > >>> on Production.
>> > > > >>>
>> > > > >>> Between I dig into furthermore and found its issue with direct
>> > > > >>> addressing mode only. When I run mco running with -T
>> mcollective or
>> > > > >>> whatever subcollectives it goes very smooth, but when I run mco
>> > > query using
>> > > > >>> flatfile or -I(direct addressing) against high number of
>> servers
>> > > only then
>> > > > >>> the issue occurs.
>> > > > >>>
>> > > > >>> So I suspect direct addressing takes time to execute and would
>> > > curious
>> > > > >>> to understand why it gives such difference. I believe direct
>> > > addressing
>> > > > >>> should be fast compare with broadcast, correct me if am wrong.
>> > > > >>>
>> > > > >>>
>> > > > >>> It’s a trade off because direct has to create and distribute a
>> > > single
>> > > > >>> message per every target. So if you have 10k targets it’s that
>> many
>> > > > >>> messages. Big pressure on activemq especially with the single
>> queue
>> > > and
>> > > > >>> processing the per node selectors
>> > > > >>>
>> > > > >>> Broadcast in turn is much lighter on the middleware and much
>> faster
>> > > all
>> > > > >>> round but you don’t know which nodes will get it
>> > > > >>>
>> > > > >>> UDP cheap and fast because it’s not guaranteed
>> > > > >>> TCP expensive and slow with many layers but guaranteed delivery
>> (or
>> > > at
>> > > > >>> least advisory about non delivery ack)
>> > > > >>>
>> > > > >>> Likewise with mco broadcast and direct.
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>>
>> > > > >>> On Sunday, January 28, 2018 at 3:11:34 PM UTC+5:30, R.I.Pienaar
>> > > wrote:
>> > > > >>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On 28 Jan 2018, at 10:35, kk21987 <kkarth...@gmail.com>
>> wrote:
>> > > > >>>>
>> > > > >>>> Some improvement with your debug help! Looks like the reply-to
>> is
>> > > > >>>> working perfect as expected. The actual problem is triggering
>> the
>> > > command
>> > > > >>>> to all the servers.
>> > > > >>>>
>> > > > >>>> I have total of 9824 servers connected with mcollective.nodes
>> and
>> > > when
>> > > > >>>> I run mco query against all the servers to execute some
>> command
>> > > then
>> > > > >>>> dispatching that mco query message all the consumers takes
>> time
>> > > which is
>> > > > >>>> why the pending messages are there and whenever the each
>> consumers
>> > > receives
>> > > > >>>> the message its slowly decreasing.
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> Great
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> Now I need to look for some tuning to get the message
>> dispatched to
>> > > all
>> > > > >>>> the consumer very quickly, so it will be good. Any suggestion
>> or
>> > > best
>> > > > >>>> tuning parameters available for this. I will try to look for
>> > > ActiveMQ
>> > > > >>>> documentation to get the something.
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> No I gave up making activemq scale this far. I have some large
>> > > rabbitmq
>> > > > >>>> clusters but that requires a ton of hardware too
>> > > > >>>>
>> > > > >>>> Choria with NATS or the Choria broker would do this
>> comfortably on
>> > > a
>> > > > >>>> single small 4GB VM with a tremendous speed improvement.
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>>
>> > > > >>>> On Sunday, January 28, 2018 at 2:03:34 PM UTC+5:30, kk21987
>> wrote:
>> > > > >>>>
>> > > > >>>>> Thanks.
>> > > > >>>>>
>> > > > >>>>> Yes I captured the client debug log as well and could see
>> that it
>> > > > >>>>> publish message with reply-to properly.
>> > > > >>>>>
>> > > > >>>>> Direct Addressing:
>> > > > >>>>>
>> > > > >>>>> D, [2018-01-28T02:24:56.851292 #17287] DEBUG -- :
>> > > activemq.rb:402:in
>> > > > >>>>> `publish' Sending a direct message to ActiveMQ target
>> > > > >>>>> '/queue/mcollective.nodes' with headers
>> > > '{"mc_identity"=>"mco-server-name",
>> > > > >>>>> "timestamp"=>"1517127896000", "expires"=>"1517127966000",
>> > > > >>>>> "reply-to"=>"/queue/data.consumer.all"}'
>> > > > >>>>>
>> > > > >>>>> Broadcast:
>> > > > >>>>>
>> > > > >>>>> D, [2018-01-28T02:25:19.345299 #17340] DEBUG -- :
>> > > activemq.rb:409:in
>> > > > >>>>> `publish' Sending a broadcast message to ActiveMQ target
>> > > > >>>>> '/topic/mcollective.shell.agent' with headers
>> > > '{"expires"=>"1517127989000",
>> > > > >>>>> "reply-to"=>"/queue/data.consumer.all",
>> > > "timestamp"=>"1517127919000"}'
>> > > > >>>>>
>> > > > >>>>> Here is the way I used to exclude the queue consumer
>> destination
>> > > on
>> > > > >>>>> Broker-A.
>> > > > >>>>>
>> > > > >>>>> <queue physicalName="data.consumer.>"/>
>> > > > >>>>>
>> > > > >>>>> I couldn't take the debug logs on ActiveMQ due to the servers
>> > > count is
>> > > > >>>>> little high, so it generate huge logs and couldn't get it
>> exactly.
>> > > But I
>> > > > >>>>> used ActiveMQ-Cli to export the JMS queue and here is a
>> example of
>> > > one
>> > > > >>>>> pending message
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> <jms-message>
>> > > > >>>>> <header>
>> > > > >>>>>
>> > > > >>>>>
>> > >
>> <message-id>ID:broker-A-16618-1517126209254-79:5:-1:1:559</message-id>
>> > > > >>>>> <delivery-mode>1</delivery-mode>
>> > > > >>>>> <destination>queue://mcollective.nodes</destination>
>> > > > >>>>> <expiration>1517127457000</expiration>
>> > > > >>>>> <priority>4</priority>
>> > > > >>>>> <redelivered>false</redelivered>
>> > > > >>>>> <reply-to>queue://data.consumer.all</reply-to>
>> > > > >>>>> <timestamp>2018-01-28T02:16:27</timestamp>
>> > > > >>>>> </header>
>> > > > >>>>> <properties>
>> > > > >>>>> <property>
>> > > > >>>>> <name>content-type</name>
>> > > > >>>>> <value>text/plain; charset=UTF-8</value>
>> > > > >>>>> </property>
>> > > > >>>>> <property>
>> > > > >>>>> <name>mc_identity</name>
>> > > > >>>>> <value>mcollective-server-name</value>
>> > > > >>>>> </property>
>> > > > >>>>> </properties>
>> > > > >>>>> </jms-message>
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>>
>> > > > >>>>> On Sunday, January 28, 2018 at 12:40:14 PM UTC+5:30,
>> R.I.Pienaar
>> > > wrote:
>> > > > >>>>>
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> On 28 Jan 2018, at 06:06, kk21987 <kkarth...@gmail.com>
>> wrote:
>> > > > >>>>>>
>> > > > >>>>>> My apologize. I misunderstood few things.
>> > > > >>>>>>
>> > > > >>>>>> Basically when I run mco query in direct addressing mode
>> then it
>> > > > >>>>>> publish that message as queue where as broadcast mode
>> publish the
>> > > message
>> > > > >>>>>> as topic. So I suspect its not an issue with the modes
>> whatever
>> > > the mco
>> > > > >>>>>> query use. The only thing I want to understand is, lets say
>> I run
>> > > mco query
>> > > > >>>>>> against 1000servers with --reply-to (fire and forget, no
>> > > acknowledgement
>> > > > >>>>>> and nothing. Correct me if am wrong), so all the response
>> message
>> > > will go
>> > > > >>>>>> to the consumer where the active subscriber is present for
>> the
>> > > given queue.
>> > > > >>>>>> The response message is not coming to Broker-A which I have
>> > > configured by
>> > > > >>>>>> starting receiver here as well and perfectly its going to
>> the
>> > > consumer
>> > > > >>>>>> where its active.
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> That sounds about right. Yes it will use a topic and all
>> nodes
>> > > will
>> > > > >>>>>> get it
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> My concern is, at the same time I see count is showing in
>> pending
>> > > > >>>>>> message for the queue "mcollective.nodes" and slowly its
>> > > decreasing(at the
>> > > > >>>>>> same enqueued and dequeued messages count also increasing
>> slowly)
>> > > which I
>> > > > >>>>>> have verified on ActiveMQ UI Jolokia monitoring (8161). So I
>> > > would like to
>> > > > >>>>>> understand why such messages count in pending messages,
>> since all
>> > > I want to
>> > > > >>>>>> just fire the command and forget and don't want to worry
>> about
>> > > response or
>> > > > >>>>>> whether its executed or not.
>> > > > >>>>>>
>> > > > >>>>>> Name ↑Number Of Pending Messages Number Of Consumers
>> Messages
>> > > > >>>>>> Enqueued Messages Dequeued Views Operations
>> > > > >>>>>> mcollective.nodes 9 84 884 875 Browse Active Consumers
>> > > > >>>>>> Active Producers
>> > > > >>>>>> Send To PurgeDelete
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> You will need to double check in the logs. Your client also
>> logs
>> > > and
>> > > > >>>>>> at debug
>> > > > >>>>>> level you can verify where the publish to
>> > > > >>>>>>
>> > > > >>>>>>
>> > > > >>>>>> On Sunday, January 28, 2018 at 7:48:15 AM UTC+5:30, kk21987
>> > > wrote:
>> > > > >>>>>>
>> > > > >>>>>>> Based on my understanding when I run mco query with
>> reply-to I
>> > > > >>>>>>> should see the reply-to queue name on the response header
>> and
>> > > its there on
>> > > > >>>>>>> mcollective server logs, which I have verified by enabling
>> debug
>> > > on the
>> > > > >>>>>>> mcollective servers.
>> > > > >>>>>>>
>> > > > >>>>>>> My worry is, I shouldn't get any response messages back to
>> the
>> > > > >>>>>>> Broker-A and all the response messages should go to the
>> consumer
>> > > server
>> > > > >>>>>>> which is running elsewhere. This is working as expected for
>> > > broadcast based
>> > > > >>>>>>> mco query but when I run direct addressing based mco query
>> then
>> > > the
>> > > > >>>>>>> response message is flowing everywhere instead of going to
>> > > consumer server
>> > > > >>>>>>> only. I have configured the exclude destination on Broker-A
>> to
>> > > make sure
>> > > > >>>>>>> none of the brokers are connected with the reply-to queue
>> name
>> > > except the
>> > > > >>>>>>> consumer server broker.
>> > > > >>>>>>>
>> > > > >>>>>>> On Sunday, January 28, 2018 at 2:08:06 AM UTC+5:30,
>> R.I.Pienaar
>> > > > >>>>>>> wrote:
>> > > > >>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>> On Sat, 27 Jan 2018, at 15:03, kk21987 wrote:
>> > > > >>>>>>>> > Hi,
>> > > > >>>>>>>> >
>> > > > >>>>>>>> > Am having some issue with --reply-to behavior and not
>> sure if
>> > > its
>> > > > >>>>>>>> an
>> > > > >>>>>>>> > issue
>> > > > >>>>>>>> > or the behavior itself like that. My setup is drawn in
>> > > > >>>>>>>> > ......
>> > > > >>>>>>>> >
>> > > > >>>>>>>> >
>> > > > >>>>>>>> > My environment is enabled with direct_addressing.
>> Basically I
>> > > > >>>>>>>> always fire
>> > > > >>>>>>>> > the mco and forget it and don't want to consume any
>> messages
>> > > on
>> > > > >>>>>>>> mco client.
>> > > > >>>>>>>> > I have configured separate consumer server where it will
>> > > consume
>> > > > >>>>>>>> the
>> > > > >>>>>>>> > messages on the queue whatever declared during the mco
>> query
>> > > and
>> > > > >>>>>>>> will get
>> > > > >>>>>>>> > result from thie consumer server. The reason for
>> configured
>> > > > >>>>>>>> separate
>> > > > >>>>>>>> > consumer server is some commands whatever I run will
>> take
>> > > more
>> > > > >>>>>>>> time to
>> > > > >>>>>>>> > execute and also doing some logic on the response
>> messages
>> > > based
>> > > > >>>>>>>> on my
>> > > > >>>>>>>> > requirement.
>> > > > >>>>>>>> >
>> > > > >>>>>>>> > I have configured exclude destination to make sure the
>> queue
>> > > > >>>>>>>> whatever am
>> > > > >>>>>>>> > using in reply-to not connected with Broker-A from other
>> > > brokers.
>> > > > >>>>>>>> >
>> > > > >>>>>>>> > When I run mco query using regex filter it pubish
>> message in
>> > > > >>>>>>>> broadcast mode
>> > > > >>>>>>>> > and consumer server consume the response without any
>> issue.
>> > > At
>> > > > >>>>>>>> the same
>> > > > >>>>>>>> > time when I check on the Broker-A ActiveMQ UI monitoring
>> I
>> > > > >>>>>>>> noticed Messages
>> > > > >>>>>>>> > Enqueued Dequeued count is NOT increased which means
>> nothing
>> > > > >>>>>>>> coming back to
>> > > > >>>>>>>> > Broker-A.
>> > > > >>>>>>>> >
>> > > > >>>>>>>> > But when I run mco query using direct addressing mode
>> (-I)
>> > > along
>> > > > >>>>>>>> with
>> > > > >>>>>>>> > reply-to the message is published as direct addressing
>> and
>> > > > >>>>>>>> consumer server
>> > > > >>>>>>>> > consume the response fine here as well. At the same time
>> when
>> > > I
>> > > > >>>>>>>> check on
>> > > > >>>>>>>> > the Broker-A ActiveMQ UI monitoring I noticed Messages
>> > > Enqueued
>> > > > >>>>>>>> Dequeued
>> > > > >>>>>>>> > count is increased which means message is coming to
>> Broker-A
>> > > as
>> > > > >>>>>>>> well at the
>> > > > >>>>>>>> > same time message is available on consumer server.
>> > > > >>>>>>>> >
>> > > > >>>>>>>> > Am not sure such difference between direct addressing
>> and
>> > > > >>>>>>>> broadcast even
>> > > > >>>>>>>> > though both use the same reply-to mechanism. I don't
>> want any
>> > > > >>>>>>>> response
>> > > > >>>>>>>> > messages come back to Broker-A and all the response
>> messages
>> > > > >>>>>>>> should go to
>> > > > >>>>>>>> > Consumer Server only. Can I have some lights on this
>> please?
>> > > > >>>>>>>>
>> > > > >>>>>>>> what you describe should work, I guess the best hints lie
>> in
>> > > the
>> > > > >>>>>>>> server debug log. Does it send a reply and does it do so
>> to
>> > > the right
>> > > > >>>>>>>> target?
>> > > > >>>>>>>>
>> > > > >>>>>>>>
>> > > > >>>>>>>> --
>> > > > >>>>>>>> R.I.Pienaar / www.devco.net / @ripienaar
>> > > > >>>>>>>>
>> > > > >>>>>>> --
>> > > > >>>>>>
>> > > > >>>>>> ---
>> > > > >>>>>> You received this message because you are subscribed to the
>> > > Google
>> > > > >>>>>> Groups "mcollective-users" group.
>> > > > >>>>>> To unsubscribe from this group and stop receiving emails
>> from it,
>> > > > >>>>>> send an email to mcollective-us...@googlegroups.com.
>> > > > >>>>>> For more options, visit https://groups.google.com/d/optout.
>> > > > >>>>>>
>> > > > >>>>>> --
>> > > > >>>>
>> > > > >>>> ---
>> > > > >>>> You received this message because you are subscribed to the
>> Google
>> > > > >>>> Groups "mcollective-users" group.
>> > > > >>>> To unsubscribe from this group and stop receiving emails from
>> it,
>> > > send
>> > > > >>>> an email to mcollective-us...@googlegroups.com.
>> > > > >>>> For more options, visit https://groups.google.com/d/optout.
>> > > > >>>>
>> > > > >>>> --
>> > > > >>>
>> > > > >>> ---
>> > > > >>> You received this message because you are subscribed to the
>> Google
>> > > > >>> Groups "mcollective-users" group.
>> > > > >>> To unsubscribe from this group and stop receiving emails from
>> it,
>> > > send
>> > > > >>> an email to mcollective-us...@googlegroups.com.
>> > > > >>> For more options, visit https://groups.google.com/d/optout.
>> > > > >>>
>> > > > >>> --
>> > > > >>
>> > > > >> ---
>> > > > >> You received this message because you are subscribed to the
>> Google
>> > > Groups
>> > > > >> "mcollective-users" group.
>> > > > >> To unsubscribe from this group and stop receiving emails from
>> it,
>> > > send an
>> > > > >> email to mcollective-us...@googlegroups.com.
>> > > > >> For more options, visit https://groups.google.com/d/optout.
>> > > > >>
>> > > > >> --
>> > > > >
>> > > > > ---
>> > > > > You received this message because you are subscribed to the
>> Google
>> > > Groups
>> > > > > "mcollective-users" group.
>> > > > > To unsubscribe from this group and stop receiving emails from it,
>> send
>> > > an
>> > > > > email to mcollective-us...@googlegroups.com <javascript:>.
>> > > > > For more options, visit https://groups.google.com/d/optout.
>> > > > >
>> > > > >
>> > > >
>> > > > --
>> > > >
>> > > > ---
>> > > > You received this message because you are subscribed to the Google
>> > > > Groups "mcollective-users" group.
>> > > > To unsubscribe from this group and stop receiving emails from it,
>> send
>> > > > an email to mcollective-us...@googlegroups.com <javascript:>.
>> > > > For more options, visit https://groups.google.com/d/optout.
>> > >
>> > >
>> > > --
>> > > R.I.Pienaar / www.devco.net / @ripienaar
>> > >
>> >
>> > --
>> >
>> > ---
>> > You received this message because you are subscribed to the Google
>> > Groups "mcollective-users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an email to mcollective-us...@googlegroups.com.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> R.I.Pienaar / www.devco.net / @ripienaar
>>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mcollective-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mcollective-us...@googlegroups.com <javascript:>.
> For more options, visit https://groups.google.com/d/optout.
>
>
--
---
You received this message because you are subscribed to the Google Groups
"mcollective-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mcollective-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
