> On 28 Jan 2018, at 13:31, kk21987 <kkarthick....@gmail.com> wrote:
>
> Yeah I agree it doesn't make sense to run without batch in direct address
> mode. But there is one problem using batch with reply-to.
>
> I have set the export for MCOLLECTIVE_EXTRA_OPTS with the values batch size
> as 200 and batch sleep as 30 and when I run mco query with --reply-to its
> returning the result on the console of mco client server itself which I don't
> want. Since I have separate consumer server configured for this.
>
> I really don't understand why it behaves like this. Any idea please.
Could be batch mode just isn’t supported in the library when using reply to. I
can look later when at a computer again
Would be trivial to write a small custom application that does this though.
>
>> On Sunday, January 28, 2018 at 3:46:04 PM UTC+5:30, R.I.Pienaar wrote:
>>
>>
>>> On 28 Jan 2018, at 11:13, kk21987 <kkarth...@gmail.com> wrote:
>>>
>>> Understood and thanks for the detailed clarification! I have already
>>> splitted as different sub collectives (2000 per subcollective) but again
>>> all the subcollectives are connected with Broker-A which will act as
>>> Central(just for naming convention) and this central will get big pressure
>>> when I run direct addressing against huge number of servers. So is there
>>> any feasibility to connect all subcollectives ActiveMQ brokers and run the
>>> command instead of using Central one by declaring some parameter in
>>> mcollective client config or using any other tool in between? I think which
>>> this way it will go smooth, since I see mcollective publish the message
>>> very quickly to Broker-A(central) and as you said this is getting big
>>> pressure.
>>
>> Why do you need to use direct addressing for your use case? If your node
>> lists arbitrary and not something you can discover against?
>>
>> You don’t use any of the confirmation features it brings?
>>
>> Only other option at this scale might be to direct address + batches
>>
>>>
>>>> On Sunday, January 28, 2018 at 3:35:01 PM UTC+5:30, R.I.Pienaar wrote:
>>>>
>>>>
>>>>> On 28 Jan 2018, at 10:48, kk21987 <kkarth...@gmail.com> wrote:
>>>>>
>>>>> I agree. I read about the Choria and very interested to use it. But am
>>>>> running little old version of Ruby, Puppet where I need to plan it for
>>>>> upgrade. But still Choria is in active dev am little concern to go ahead
>>>>> it on Production.
>>>>>
>>>>> Between I dig into furthermore and found its issue with direct addressing
>>>>> mode only. When I run mco running with -T mcollective or whatever
>>>>> subcollectives it goes very smooth, but when I run mco query using
>>>>> flatfile or -I(direct addressing) against high number of servers only
>>>>> then the issue occurs.
>>>>>
>>>>> So I suspect direct addressing takes time to execute and would curious to
>>>>> understand why it gives such difference. I believe direct addressing
>>>>> should be fast compare with broadcast, correct me if am wrong.
>>>>
>>>> It’s a trade off because direct has to create and distribute a single
>>>> message per every target. So if you have 10k targets it’s that many
>>>> messages. Big pressure on activemq especially with the single queue and
>>>> processing the per node selectors
>>>>
>>>> Broadcast in turn is much lighter on the middleware and much faster all
>>>> round but you don’t know which nodes will get it
>>>>
>>>> UDP cheap and fast because it’s not guaranteed
>>>> TCP expensive and slow with many layers but guaranteed delivery (or at
>>>> least advisory about non delivery ack)
>>>>
>>>> Likewise with mco broadcast and direct.
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On Sunday, January 28, 2018 at 3:11:34 PM UTC+5:30, R.I.Pienaar wrote:
>>>>>>
>>>>>>
>>>>>>> On 28 Jan 2018, at 10:35, kk21987 <kkarth...@gmail.com> wrote:
>>>>>>>
>>>>>>> Some improvement with your debug help! Looks like the reply-to is
>>>>>>> working perfect as expected. The actual problem is triggering the
>>>>>>> command to all the servers.
>>>>>>>
>>>>>>> I have total of 9824 servers connected with mcollective.nodes and when
>>>>>>> I run mco query against all the servers to execute some command then
>>>>>>> dispatching that mco query message all the consumers takes time which
>>>>>>> is why the pending messages are there and whenever the each consumers
>>>>>>> receives the message its slowly decreasing.
>>>>>>
>>>>>> Great
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Now I need to look for some tuning to get the message dispatched to all
>>>>>>> the consumer very quickly, so it will be good. Any suggestion or best
>>>>>>> tuning parameters available for this. I will try to look for ActiveMQ
>>>>>>> documentation to get the something.
>>>>>>
>>>>>> No I gave up making activemq scale this far. I have some large rabbitmq
>>>>>> clusters but that requires a ton of hardware too
>>>>>>
>>>>>> Choria with NATS or the Choria broker would do this comfortably on a
>>>>>> single small 4GB VM with a tremendous speed improvement.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Sunday, January 28, 2018 at 2:03:34 PM UTC+5:30, kk21987 wrote:
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Yes I captured the client debug log as well and could see that it
>>>>>>>> publish message with reply-to properly.
>>>>>>>>
>>>>>>>> Direct Addressing:
>>>>>>>>
>>>>>>>> D, [2018-01-28T02:24:56.851292 #17287] DEBUG -- : activemq.rb:402:in
>>>>>>>> `publish' Sending a direct message to ActiveMQ target
>>>>>>>> '/queue/mcollective.nodes' with headers
>>>>>>>> '{"mc_identity"=>"mco-server-name", "timestamp"=>"1517127896000",
>>>>>>>> "expires"=>"1517127966000", "reply-to"=>"/queue/data.consumer.all"}'
>>>>>>>>
>>>>>>>> Broadcast:
>>>>>>>>
>>>>>>>> D, [2018-01-28T02:25:19.345299 #17340] DEBUG -- : activemq.rb:409:in
>>>>>>>> `publish' Sending a broadcast message to ActiveMQ target
>>>>>>>> '/topic/mcollective.shell.agent' with headers
>>>>>>>> '{"expires"=>"1517127989000", "reply-to"=>"/queue/data.consumer.all",
>>>>>>>> "timestamp"=>"1517127919000"}'
>>>>>>>>
>>>>>>>> Here is the way I used to exclude the queue consumer destination on
>>>>>>>> Broker-A.
>>>>>>>>
>>>>>>>> <queue physicalName="data.consumer.>"/>
>>>>>>>>
>>>>>>>> I couldn't take the debug logs on ActiveMQ due to the servers count is
>>>>>>>> little high, so it generate huge logs and couldn't get it exactly. But
>>>>>>>> I used ActiveMQ-Cli to export the JMS queue and here is a example of
>>>>>>>> one pending message
>>>>>>>>
>>>>>>>>
>>>>>>>> <jms-message>
>>>>>>>> <header>
>>>>>>>>
>>>>>>>> <message-id>ID:broker-A-16618-1517126209254-79:5:-1:1:559</message-id>
>>>>>>>> <delivery-mode>1</delivery-mode>
>>>>>>>> <destination>queue://mcollective.nodes</destination>
>>>>>>>> <expiration>1517127457000</expiration>
>>>>>>>> <priority>4</priority>
>>>>>>>> <redelivered>false</redelivered>
>>>>>>>> <reply-to>queue://data.consumer.all</reply-to>
>>>>>>>> <timestamp>2018-01-28T02:16:27</timestamp>
>>>>>>>> </header>
>>>>>>>> <properties>
>>>>>>>> <property>
>>>>>>>> <name>content-type</name>
>>>>>>>> <value>text/plain; charset=UTF-8</value>
>>>>>>>> </property>
>>>>>>>> <property>
>>>>>>>> <name>mc_identity</name>
>>>>>>>> <value>mcollective-server-name</value>
>>>>>>>> </property>
>>>>>>>> </properties>
>>>>>>>> </jms-message>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Sunday, January 28, 2018 at 12:40:14 PM UTC+5:30, R.I.Pienaar
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 28 Jan 2018, at 06:06, kk21987 <kkarth...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> My apologize. I misunderstood few things.
>>>>>>>>>>
>>>>>>>>>> Basically when I run mco query in direct addressing mode then it
>>>>>>>>>> publish that message as queue where as broadcast mode publish the
>>>>>>>>>> message as topic. So I suspect its not an issue with the modes
>>>>>>>>>> whatever the mco query use. The only thing I want to understand is,
>>>>>>>>>> lets say I run mco query against 1000servers with --reply-to (fire
>>>>>>>>>> and forget, no acknowledgement and nothing. Correct me if am wrong),
>>>>>>>>>> so all the response message will go to the consumer where the active
>>>>>>>>>> subscriber is present for the given queue. The response message is
>>>>>>>>>> not coming to Broker-A which I have configured by starting receiver
>>>>>>>>>> here as well and perfectly its going to the consumer where its
>>>>>>>>>> active.
>>>>>>>>>
>>>>>>>>> That sounds about right. Yes it will use a topic and all nodes will
>>>>>>>>> get it
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> My concern is, at the same time I see count is showing in pending
>>>>>>>>>> message for the queue "mcollective.nodes" and slowly its
>>>>>>>>>> decreasing(at the same enqueued and dequeued messages count also
>>>>>>>>>> increasing slowly) which I have verified on ActiveMQ UI Jolokia
>>>>>>>>>> monitoring (8161). So I would like to understand why such messages
>>>>>>>>>> count in pending messages, since all I want to just fire the command
>>>>>>>>>> and forget and don't want to worry about response or whether its
>>>>>>>>>> executed or not.
>>>>>>>>>>
>>>>>>>>>> Name ↑ Number Of Pending Messages Number Of Consumers
>>>>>>>>>> Messages Enqueued Messages Dequeued Views Operations
>>>>>>>>>> mcollective.nodes 9 84 884 875 Browse Active
>>>>>>>>>> Consumers
>>>>>>>>>> Active Producers
>>>>>>>>>> Send To PurgeDelete
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> You will need to double check in the logs. Your client also logs and
>>>>>>>>> at debug
>>>>>>>>> level you can verify where the publish to
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> On Sunday, January 28, 2018 at 7:48:15 AM UTC+5:30, kk21987 wrote:
>>>>>>>>>>> Based on my understanding when I run mco query with reply-to I
>>>>>>>>>>> should see the reply-to queue name on the response header and its
>>>>>>>>>>> there on mcollective server logs, which I have verified by enabling
>>>>>>>>>>> debug on the mcollective servers.
>>>>>>>>>>>
>>>>>>>>>>> My worry is, I shouldn't get any response messages back to the
>>>>>>>>>>> Broker-A and all the response messages should go to the consumer
>>>>>>>>>>> server which is running elsewhere. This is working as expected for
>>>>>>>>>>> broadcast based mco query but when I run direct addressing based
>>>>>>>>>>> mco query then the response message is flowing everywhere instead
>>>>>>>>>>> of going to consumer server only. I have configured the exclude
>>>>>>>>>>> destination on Broker-A to make sure none of the brokers are
>>>>>>>>>>> connected with the reply-to queue name except the consumer server
>>>>>>>>>>> broker.
>>>>>>>>>>>
>>>>>>>>>>>> On Sunday, January 28, 2018 at 2:08:06 AM UTC+5:30, R.I.Pienaar
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, 27 Jan 2018, at 15:03, kk21987 wrote:
>>>>>>>>>>>> > Hi,
>>>>>>>>>>>> >
>>>>>>>>>>>> > Am having some issue with --reply-to behavior and not sure if
>>>>>>>>>>>> > its an
>>>>>>>>>>>> > issue
>>>>>>>>>>>> > or the behavior itself like that. My setup is drawn in
>>>>>>>>>>>> > ......
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > My environment is enabled with direct_addressing. Basically I
>>>>>>>>>>>> > always fire
>>>>>>>>>>>> > the mco and forget it and don't want to consume any messages on
>>>>>>>>>>>> > mco client.
>>>>>>>>>>>> > I have configured separate consumer server where it will consume
>>>>>>>>>>>> > the
>>>>>>>>>>>> > messages on the queue whatever declared during the mco query and
>>>>>>>>>>>> > will get
>>>>>>>>>>>> > result from thie consumer server. The reason for configured
>>>>>>>>>>>> > separate
>>>>>>>>>>>> > consumer server is some commands whatever I run will take more
>>>>>>>>>>>> > time to
>>>>>>>>>>>> > execute and also doing some logic on the response messages based
>>>>>>>>>>>> > on my
>>>>>>>>>>>> > requirement.
>>>>>>>>>>>> >
>>>>>>>>>>>> > I have configured exclude destination to make sure the queue
>>>>>>>>>>>> > whatever am
>>>>>>>>>>>> > using in reply-to not connected with Broker-A from other
>>>>>>>>>>>> > brokers.
>>>>>>>>>>>> >
>>>>>>>>>>>> > When I run mco query using regex filter it pubish message in
>>>>>>>>>>>> > broadcast mode
>>>>>>>>>>>> > and consumer server consume the response without any issue. At
>>>>>>>>>>>> > the same
>>>>>>>>>>>> > time when I check on the Broker-A ActiveMQ UI monitoring I
>>>>>>>>>>>> > noticed Messages
>>>>>>>>>>>> > Enqueued Dequeued count is NOT increased which means nothing
>>>>>>>>>>>> > coming back to
>>>>>>>>>>>> > Broker-A.
>>>>>>>>>>>> >
>>>>>>>>>>>> > But when I run mco query using direct addressing mode (-I) along
>>>>>>>>>>>> > with
>>>>>>>>>>>> > reply-to the message is published as direct addressing and
>>>>>>>>>>>> > consumer server
>>>>>>>>>>>> > consume the response fine here as well. At the same time when I
>>>>>>>>>>>> > check on
>>>>>>>>>>>> > the Broker-A ActiveMQ UI monitoring I noticed Messages Enqueued
>>>>>>>>>>>> > Dequeued
>>>>>>>>>>>> > count is increased which means message is coming to Broker-A as
>>>>>>>>>>>> > well at the
>>>>>>>>>>>> > same time message is available on consumer server.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Am not sure such difference between direct addressing and
>>>>>>>>>>>> > broadcast even
>>>>>>>>>>>> > though both use the same reply-to mechanism. I don't want any
>>>>>>>>>>>> > response
>>>>>>>>>>>> > messages come back to Broker-A and all the response messages
>>>>>>>>>>>> > should go to
>>>>>>>>>>>> > Consumer Server only. Can I have some lights on this please?
>>>>>>>>>>>>
>>>>>>>>>>>> what you describe should work, I guess the best hints lie in the
>>>>>>>>>>>> server debug log. Does it send a reply and does it do so to the
>>>>>>>>>>>> right target?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> R.I.Pienaar / www.devco.net / @ripienaar
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>>> Groups "mcollective-users" group.
>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>> send an email to mcollective-us...@googlegroups.com.
>>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "mcollective-users" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>>> an email to mcollective-us...@googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>> --
>>>>>
>>>>> ---
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "mcollective-users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to mcollective-us...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "mcollective-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to mcollective-us...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mcollective-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mcollective-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups
"mcollective-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mcollective-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
