> On 28 Jan 2018, at 10:48, kk21987 <kkarthick....@gmail.com> wrote:
>
> I agree. I read about the Choria and very interested to use it. But am
> running little old version of Ruby, Puppet where I need to plan it for
> upgrade. But still Choria is in active dev am little concern to go ahead it
> on Production.
>
> Between I dig into furthermore and found its issue with direct addressing
> mode only. When I run mco running with -T mcollective or whatever
> subcollectives it goes very smooth, but when I run mco query using flatfile
> or -I(direct addressing) against high number of servers only then the issue
> occurs.
>
> So I suspect direct addressing takes time to execute and would curious to
> understand why it gives such difference. I believe direct addressing should
> be fast compare with broadcast, correct me if am wrong.
It’s a trade off because direct has to create and distribute a single message
per every target. So if you have 10k targets it’s that many messages. Big
pressure on activemq especially with the single queue and processing the per
node selectors
Broadcast in turn is much lighter on the middleware and much faster all round
but you don’t know which nodes will get it
UDP cheap and fast because it’s not guaranteed
TCP expensive and slow with many layers but guaranteed delivery (or at least
advisory about non delivery ack)
Likewise with mco broadcast and direct.
>
>
>
>
>> On Sunday, January 28, 2018 at 3:11:34 PM UTC+5:30, R.I.Pienaar wrote:
>>
>>
>>> On 28 Jan 2018, at 10:35, kk21987 <kkarth...@gmail.com> wrote:
>>>
>>> Some improvement with your debug help! Looks like the reply-to is working
>>> perfect as expected. The actual problem is triggering the command to all
>>> the servers.
>>>
>>> I have total of 9824 servers connected with mcollective.nodes and when I
>>> run mco query against all the servers to execute some command then
>>> dispatching that mco query message all the consumers takes time which is
>>> why the pending messages are there and whenever the each consumers receives
>>> the message its slowly decreasing.
>>
>> Great
>>
>>
>>>
>>> Now I need to look for some tuning to get the message dispatched to all the
>>> consumer very quickly, so it will be good. Any suggestion or best tuning
>>> parameters available for this. I will try to look for ActiveMQ
>>> documentation to get the something.
>>
>> No I gave up making activemq scale this far. I have some large rabbitmq
>> clusters but that requires a ton of hardware too
>>
>> Choria with NATS or the Choria broker would do this comfortably on a single
>> small 4GB VM with a tremendous speed improvement.
>>
>>
>>>
>>>
>>>
>>>> On Sunday, January 28, 2018 at 2:03:34 PM UTC+5:30, kk21987 wrote:
>>>> Thanks.
>>>>
>>>> Yes I captured the client debug log as well and could see that it publish
>>>> message with reply-to properly.
>>>>
>>>> Direct Addressing:
>>>>
>>>> D, [2018-01-28T02:24:56.851292 #17287] DEBUG -- : activemq.rb:402:in
>>>> `publish' Sending a direct message to ActiveMQ target
>>>> '/queue/mcollective.nodes' with headers
>>>> '{"mc_identity"=>"mco-server-name", "timestamp"=>"1517127896000",
>>>> "expires"=>"1517127966000", "reply-to"=>"/queue/data.consumer.all"}'
>>>>
>>>> Broadcast:
>>>>
>>>> D, [2018-01-28T02:25:19.345299 #17340] DEBUG -- : activemq.rb:409:in
>>>> `publish' Sending a broadcast message to ActiveMQ target
>>>> '/topic/mcollective.shell.agent' with headers
>>>> '{"expires"=>"1517127989000", "reply-to"=>"/queue/data.consumer.all",
>>>> "timestamp"=>"1517127919000"}'
>>>>
>>>> Here is the way I used to exclude the queue consumer destination on
>>>> Broker-A.
>>>>
>>>> <queue physicalName="data.consumer.>"/>
>>>>
>>>> I couldn't take the debug logs on ActiveMQ due to the servers count is
>>>> little high, so it generate huge logs and couldn't get it exactly. But I
>>>> used ActiveMQ-Cli to export the JMS queue and here is a example of one
>>>> pending message
>>>>
>>>>
>>>> <jms-message>
>>>> <header>
>>>>
>>>> <message-id>ID:broker-A-16618-1517126209254-79:5:-1:1:559</message-id>
>>>> <delivery-mode>1</delivery-mode>
>>>> <destination>queue://mcollective.nodes</destination>
>>>> <expiration>1517127457000</expiration>
>>>> <priority>4</priority>
>>>> <redelivered>false</redelivered>
>>>> <reply-to>queue://data.consumer.all</reply-to>
>>>> <timestamp>2018-01-28T02:16:27</timestamp>
>>>> </header>
>>>> <properties>
>>>> <property>
>>>> <name>content-type</name>
>>>> <value>text/plain; charset=UTF-8</value>
>>>> </property>
>>>> <property>
>>>> <name>mc_identity</name>
>>>> <value>mcollective-server-name</value>
>>>> </property>
>>>> </properties>
>>>> </jms-message>
>>>>
>>>>
>>>>
>>>>
>>>>> On Sunday, January 28, 2018 at 12:40:14 PM UTC+5:30, R.I.Pienaar wrote:
>>>>>
>>>>>
>>>>>> On 28 Jan 2018, at 06:06, kk21987 <kkarth...@gmail.com> wrote:
>>>>>>
>>>>>> My apologize. I misunderstood few things.
>>>>>>
>>>>>> Basically when I run mco query in direct addressing mode then it publish
>>>>>> that message as queue where as broadcast mode publish the message as
>>>>>> topic. So I suspect its not an issue with the modes whatever the mco
>>>>>> query use. The only thing I want to understand is, lets say I run mco
>>>>>> query against 1000servers with --reply-to (fire and forget, no
>>>>>> acknowledgement and nothing. Correct me if am wrong), so all the
>>>>>> response message will go to the consumer where the active subscriber is
>>>>>> present for the given queue. The response message is not coming to
>>>>>> Broker-A which I have configured by starting receiver here as well and
>>>>>> perfectly its going to the consumer where its active.
>>>>>
>>>>> That sounds about right. Yes it will use a topic and all nodes will get it
>>>>>
>>>>>>
>>>>>> My concern is, at the same time I see count is showing in pending
>>>>>> message for the queue "mcollective.nodes" and slowly its decreasing(at
>>>>>> the same enqueued and dequeued messages count also increasing slowly)
>>>>>> which I have verified on ActiveMQ UI Jolokia monitoring (8161). So I
>>>>>> would like to understand why such messages count in pending messages,
>>>>>> since all I want to just fire the command and forget and don't want to
>>>>>> worry about response or whether its executed or not.
>>>>>>
>>>>>> Name ↑ Number Of Pending Messages Number Of Consumers
>>>>>> Messages Enqueued Messages Dequeued Views Operations
>>>>>> mcollective.nodes 9 84 884 875 Browse Active
>>>>>> Consumers
>>>>>> Active Producers
>>>>>> Send To PurgeDelete
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> You will need to double check in the logs. Your client also logs and at
>>>>> debug
>>>>> level you can verify where the publish to
>>>>>
>>>>>
>>>>>>> On Sunday, January 28, 2018 at 7:48:15 AM UTC+5:30, kk21987 wrote:
>>>>>>> Based on my understanding when I run mco query with reply-to I should
>>>>>>> see the reply-to queue name on the response header and its there on
>>>>>>> mcollective server logs, which I have verified by enabling debug on the
>>>>>>> mcollective servers.
>>>>>>>
>>>>>>> My worry is, I shouldn't get any response messages back to the Broker-A
>>>>>>> and all the response messages should go to the consumer server which is
>>>>>>> running elsewhere. This is working as expected for broadcast based mco
>>>>>>> query but when I run direct addressing based mco query then the
>>>>>>> response message is flowing everywhere instead of going to consumer
>>>>>>> server only. I have configured the exclude destination on Broker-A to
>>>>>>> make sure none of the brokers are connected with the reply-to queue
>>>>>>> name except the consumer server broker.
>>>>>>>
>>>>>>>> On Sunday, January 28, 2018 at 2:08:06 AM UTC+5:30, R.I.Pienaar wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, 27 Jan 2018, at 15:03, kk21987 wrote:
>>>>>>>> > Hi,
>>>>>>>> >
>>>>>>>> > Am having some issue with --reply-to behavior and not sure if its an
>>>>>>>> > issue
>>>>>>>> > or the behavior itself like that. My setup is drawn in
>>>>>>>> > ......
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > My environment is enabled with direct_addressing. Basically I always
>>>>>>>> > fire
>>>>>>>> > the mco and forget it and don't want to consume any messages on mco
>>>>>>>> > client.
>>>>>>>> > I have configured separate consumer server where it will consume the
>>>>>>>> > messages on the queue whatever declared during the mco query and
>>>>>>>> > will get
>>>>>>>> > result from thie consumer server. The reason for configured separate
>>>>>>>> > consumer server is some commands whatever I run will take more time
>>>>>>>> > to
>>>>>>>> > execute and also doing some logic on the response messages based on
>>>>>>>> > my
>>>>>>>> > requirement.
>>>>>>>> >
>>>>>>>> > I have configured exclude destination to make sure the queue
>>>>>>>> > whatever am
>>>>>>>> > using in reply-to not connected with Broker-A from other brokers.
>>>>>>>> >
>>>>>>>> > When I run mco query using regex filter it pubish message in
>>>>>>>> > broadcast mode
>>>>>>>> > and consumer server consume the response without any issue. At the
>>>>>>>> > same
>>>>>>>> > time when I check on the Broker-A ActiveMQ UI monitoring I noticed
>>>>>>>> > Messages
>>>>>>>> > Enqueued Dequeued count is NOT increased which means nothing coming
>>>>>>>> > back to
>>>>>>>> > Broker-A.
>>>>>>>> >
>>>>>>>> > But when I run mco query using direct addressing mode (-I) along
>>>>>>>> > with
>>>>>>>> > reply-to the message is published as direct addressing and consumer
>>>>>>>> > server
>>>>>>>> > consume the response fine here as well. At the same time when I
>>>>>>>> > check on
>>>>>>>> > the Broker-A ActiveMQ UI monitoring I noticed Messages Enqueued
>>>>>>>> > Dequeued
>>>>>>>> > count is increased which means message is coming to Broker-A as well
>>>>>>>> > at the
>>>>>>>> > same time message is available on consumer server.
>>>>>>>> >
>>>>>>>> > Am not sure such difference between direct addressing and broadcast
>>>>>>>> > even
>>>>>>>> > though both use the same reply-to mechanism. I don't want any
>>>>>>>> > response
>>>>>>>> > messages come back to Broker-A and all the response messages should
>>>>>>>> > go to
>>>>>>>> > Consumer Server only. Can I have some lights on this please?
>>>>>>>>
>>>>>>>> what you describe should work, I guess the best hints lie in the
>>>>>>>> server debug log. Does it send a reply and does it do so to the right
>>>>>>>> target?
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> R.I.Pienaar / www.devco.net / @ripienaar
>>>>>>
>>>>>> --
>>>>>>
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "mcollective-users" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to mcollective-us...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google Groups
>>> "mcollective-users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to mcollective-us...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mcollective-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mcollective-users+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
---
You received this message because you are subscribed to the Google Groups
"mcollective-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mcollective-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
