Great and thanks! Let me try this one, between in which location I should place this file to make normal mco query work with batch.
On Sunday, January 28, 2018 at 9:49:53 PM UTC+5:30, R.I.Pienaar wrote: > here's a batch fire and forget client, of course it doesnt return > instantly but it doesnt expect replies or wait for replies > > https://gist.github.com/ripienaar/52e73db531cc22b434b8433ababd8df2 > > On Sun, 28 Jan 2018, at 16:51, kk21987 wrote: > > It would be good if I get true batch mode where all requests have the > same > > ID, since my consumer segregates the response output based on the ID > only. > > > > The reason why I want to go with batch mode is, I have some set of > servers > > where I need to collect some data. But in this the data and also the > > servers list will keep vary which comes from different subcollectives. > So I > > don't have specific facts filter to run in broadcast mode and also I > can't > > make such facts also. Its basically kind of requirement for us to run > this > > often. > > > > > > On Sunday, January 28, 2018 at 9:14:52 PM UTC+5:30, R.I.Pienaar wrote: > > > > > > > > > > > On Sun, 28 Jan 2018, at 15:29, kk21987 wrote: > > > > Yes please. If the reply-to works well with batch mode it would be > great > > > > for me as it will solve my proble :) > > > > > > yeah fire and forget doesnt support batch mode in the normal client > > > > > > in your use case does it matter if the replies all have the same > request > > > id - ie. do you want a true batch mode where all requests have the > same ID > > > or many seperate requests would be fine? > > > > > > I still am curious why you need to use direct mode rather than > broadcast > > > > > > > > On Sunday, January 28, 2018 at 6:23:35 PM UTC+5:30, R.I.Pienaar > wrote: > > > > > > > > > > > > > > > On 28 Jan 2018, at 13:31, kk21987 <kkarth...@gmail.com > <javascript:>> > > > > > wrote: > > > > > > > > > > Yeah I agree it doesn't make sense to run without batch in direct > > > address > > > > > mode. But there is one problem using batch with reply-to. > > > > > > > > > > I have set the export for MCOLLECTIVE_EXTRA_OPTS with the values > batch > > > > > size as 200 and batch sleep as 30 and when I run mco query with > > > --reply-to > > > > > its returning the result on the console of mco client server > itself > > > which I > > > > > don't want. Since I have separate consumer server configured for > this. > > > > > > > > > > I really don't understand why it behaves like this. Any idea > please. > > > > > > > > > > > > > > > Could be batch mode just isn’t supported in the library when using > > > reply > > > > > to. I can look later when at a computer again > > > > > > > > > > Would be trivial to write a small custom application that does > this > > > > > though. > > > > > > > > > > > > > > > On Sunday, January 28, 2018 at 3:46:04 PM UTC+5:30, R.I.Pienaar > wrote: > > > > > > > > > >> > > > > >> > > > > >> On 28 Jan 2018, at 11:13, kk21987 <kkarth...@gmail.com> wrote: > > > > >> > > > > >> Understood and thanks for the detailed clarification! I have > already > > > > >> splitted as different sub collectives (2000 per subcollective) > but > > > again > > > > >> all the subcollectives are connected with Broker-A which will act > as > > > > >> Central(just for naming convention) and this central will get big > > > pressure > > > > >> when I run direct addressing against huge number of servers. So > is > > > there > > > > >> any feasibility to connect all subcollectives ActiveMQ brokers > and > > > run the > > > > >> command instead of using Central one by declaring some parameter > in > > > > >> mcollective client config or using any other tool in between? I > think > > > which > > > > >> this way it will go smooth, since I see mcollective publish the > > > message > > > > >> very quickly to Broker-A(central) and as you said this is getting > big > > > > >> pressure. > > > > >> > > > > >> > > > > >> Why do you need to use direct addressing for your use case? If > your > > > node > > > > >> lists arbitrary and not something you can discover against? > > > > >> > > > > >> You don’t use any of the confirmation features it brings? > > > > >> > > > > >> Only other option at this scale might be to direct address + > batches > > > > >> > > > > >> > > > > >> On Sunday, January 28, 2018 at 3:35:01 PM UTC+5:30, R.I.Pienaar > > > wrote: > > > > >>> > > > > >>> > > > > >>> > > > > >>> On 28 Jan 2018, at 10:48, kk21987 <kkarth...@gmail.com> wrote: > > > > >>> > > > > >>> I agree. I read about the Choria and very interested to use it. > But > > > am > > > > >>> running little old version of Ruby, Puppet where I need to plan > it > > > for > > > > >>> upgrade. But still Choria is in active dev am little concern to > go > > > ahead it > > > > >>> on Production. > > > > >>> > > > > >>> Between I dig into furthermore and found its issue with direct > > > > >>> addressing mode only. When I run mco running with -T mcollective > or > > > > >>> whatever subcollectives it goes very smooth, but when I run mco > > > query using > > > > >>> flatfile or -I(direct addressing) against high number of servers > > > only then > > > > >>> the issue occurs. > > > > >>> > > > > >>> So I suspect direct addressing takes time to execute and would > > > curious > > > > >>> to understand why it gives such difference. I believe direct > > > addressing > > > > >>> should be fast compare with broadcast, correct me if am wrong. > > > > >>> > > > > >>> > > > > >>> It’s a trade off because direct has to create and distribute a > > > single > > > > >>> message per every target. So if you have 10k targets it’s that > many > > > > >>> messages. Big pressure on activemq especially with the single > queue > > > and > > > > >>> processing the per node selectors > > > > >>> > > > > >>> Broadcast in turn is much lighter on the middleware and much > faster > > > all > > > > >>> round but you don’t know which nodes will get it > > > > >>> > > > > >>> UDP cheap and fast because it’s not guaranteed > > > > >>> TCP expensive and slow with many layers but guaranteed delivery > (or > > > at > > > > >>> least advisory about non delivery ack) > > > > >>> > > > > >>> Likewise with mco broadcast and direct. > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> > > > > >>> On Sunday, January 28, 2018 at 3:11:34 PM UTC+5:30, R.I.Pienaar > > > wrote: > > > > >>> > > > > >>>> > > > > >>>> > > > > >>>> On 28 Jan 2018, at 10:35, kk21987 <kkarth...@gmail.com> wrote: > > > > >>>> > > > > >>>> Some improvement with your debug help! Looks like the reply-to > is > > > > >>>> working perfect as expected. The actual problem is triggering > the > > > command > > > > >>>> to all the servers. > > > > >>>> > > > > >>>> I have total of 9824 servers connected with mcollective.nodes > and > > > when > > > > >>>> I run mco query against all the servers to execute some command > > > then > > > > >>>> dispatching that mco query message all the consumers takes time > > > which is > > > > >>>> why the pending messages are there and whenever the each > consumers > > > receives > > > > >>>> the message its slowly decreasing. > > > > >>>> > > > > >>>> > > > > >>>> Great > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> Now I need to look for some tuning to get the message > dispatched to > > > all > > > > >>>> the consumer very quickly, so it will be good. Any suggestion > or > > > best > > > > >>>> tuning parameters available for this. I will try to look for > > > ActiveMQ > > > > >>>> documentation to get the something. > > > > >>>> > > > > >>>> > > > > >>>> No I gave up making activemq scale this far. I have some large > > > rabbitmq > > > > >>>> clusters but that requires a ton of hardware too > > > > >>>> > > > > >>>> Choria with NATS or the Choria broker would do this comfortably > on > > > a > > > > >>>> single small 4GB VM with a tremendous speed improvement. > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> On Sunday, January 28, 2018 at 2:03:34 PM UTC+5:30, kk21987 > wrote: > > > > >>>> > > > > >>>>> Thanks. > > > > >>>>> > > > > >>>>> Yes I captured the client debug log as well and could see that > it > > > > >>>>> publish message with reply-to properly. > > > > >>>>> > > > > >>>>> Direct Addressing: > > > > >>>>> > > > > >>>>> D, [2018-01-28T02:24:56.851292 #17287] DEBUG -- : > > > activemq.rb:402:in > > > > >>>>> `publish' Sending a direct message to ActiveMQ target > > > > >>>>> '/queue/mcollective.nodes' with headers > > > '{"mc_identity"=>"mco-server-name", > > > > >>>>> "timestamp"=>"1517127896000", "expires"=>"1517127966000", > > > > >>>>> "reply-to"=>"/queue/data.consumer.all"}' > > > > >>>>> > > > > >>>>> Broadcast: > > > > >>>>> > > > > >>>>> D, [2018-01-28T02:25:19.345299 #17340] DEBUG -- : > > > activemq.rb:409:in > > > > >>>>> `publish' Sending a broadcast message to ActiveMQ target > > > > >>>>> '/topic/mcollective.shell.agent' with headers > > > '{"expires"=>"1517127989000", > > > > >>>>> "reply-to"=>"/queue/data.consumer.all", > > > "timestamp"=>"1517127919000"}' > > > > >>>>> > > > > >>>>> Here is the way I used to exclude the queue consumer > destination > > > on > > > > >>>>> Broker-A. > > > > >>>>> > > > > >>>>> <queue physicalName="data.consumer.>"/> > > > > >>>>> > > > > >>>>> I couldn't take the debug logs on ActiveMQ due to the servers > > > count is > > > > >>>>> little high, so it generate huge logs and couldn't get it > exactly. > > > But I > > > > >>>>> used ActiveMQ-Cli to export the JMS queue and here is a > example of > > > one > > > > >>>>> pending message > > > > >>>>> > > > > >>>>> > > > > >>>>> <jms-message> > > > > >>>>> <header> > > > > >>>>> > > > > >>>>> > > > <message-id>ID:broker-A-16618-1517126209254-79:5:-1:1:559</message-id> > > > > >>>>> <delivery-mode>1</delivery-mode> > > > > >>>>> <destination>queue://mcollective.nodes</destination> > > > > >>>>> <expiration>1517127457000</expiration> > > > > >>>>> <priority>4</priority> > > > > >>>>> <redelivered>false</redelivered> > > > > >>>>> <reply-to>queue://data.consumer.all</reply-to> > > > > >>>>> <timestamp>2018-01-28T02:16:27</timestamp> > > > > >>>>> </header> > > > > >>>>> <properties> > > > > >>>>> <property> > > > > >>>>> <name>content-type</name> > > > > >>>>> <value>text/plain; charset=UTF-8</value> > > > > >>>>> </property> > > > > >>>>> <property> > > > > >>>>> <name>mc_identity</name> > > > > >>>>> <value>mcollective-server-name</value> > > > > >>>>> </property> > > > > >>>>> </properties> > > > > >>>>> </jms-message> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> On Sunday, January 28, 2018 at 12:40:14 PM UTC+5:30, > R.I.Pienaar > > > wrote: > > > > >>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> On 28 Jan 2018, at 06:06, kk21987 <kkarth...@gmail.com> > wrote: > > > > >>>>>> > > > > >>>>>> My apologize. I misunderstood few things. > > > > >>>>>> > > > > >>>>>> Basically when I run mco query in direct addressing mode then > it > > > > >>>>>> publish that message as queue where as broadcast mode publish > the > > > message > > > > >>>>>> as topic. So I suspect its not an issue with the modes > whatever > > > the mco > > > > >>>>>> query use. The only thing I want to understand is, lets say I > run > > > mco query > > > > >>>>>> against 1000servers with --reply-to (fire and forget, no > > > acknowledgement > > > > >>>>>> and nothing. Correct me if am wrong), so all the response > message > > > will go > > > > >>>>>> to the consumer where the active subscriber is present for > the > > > given queue. > > > > >>>>>> The response message is not coming to Broker-A which I have > > > configured by > > > > >>>>>> starting receiver here as well and perfectly its going to the > > > consumer > > > > >>>>>> where its active. > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> That sounds about right. Yes it will use a topic and all > nodes > > > will > > > > >>>>>> get it > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> My concern is, at the same time I see count is showing in > pending > > > > >>>>>> message for the queue "mcollective.nodes" and slowly its > > > decreasing(at the > > > > >>>>>> same enqueued and dequeued messages count also increasing > slowly) > > > which I > > > > >>>>>> have verified on ActiveMQ UI Jolokia monitoring (8161). So I > > > would like to > > > > >>>>>> understand why such messages count in pending messages, since > all > > > I want to > > > > >>>>>> just fire the command and forget and don't want to worry > about > > > response or > > > > >>>>>> whether its executed or not. > > > > >>>>>> > > > > >>>>>> Name ↑Number Of Pending Messages Number Of Consumers > Messages > > > > >>>>>> Enqueued Messages Dequeued Views Operations > > > > >>>>>> mcollective.nodes 9 84 884 875 Browse Active Consumers > > > > >>>>>> Active Producers > > > > >>>>>> Send To PurgeDelete > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> You will need to double check in the logs. Your client also > logs > > > and > > > > >>>>>> at debug > > > > >>>>>> level you can verify where the publish to > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> On Sunday, January 28, 2018 at 7:48:15 AM UTC+5:30, kk21987 > > > wrote: > > > > >>>>>> > > > > >>>>>>> Based on my understanding when I run mco query with reply-to > I > > > > >>>>>>> should see the reply-to queue name on the response header > and > > > its there on > > > > >>>>>>> mcollective server logs, which I have verified by enabling > debug > > > on the > > > > >>>>>>> mcollective servers. > > > > >>>>>>> > > > > >>>>>>> My worry is, I shouldn't get any response messages back to > the > > > > >>>>>>> Broker-A and all the response messages should go to the > consumer > > > server > > > > >>>>>>> which is running elsewhere. This is working as expected for > > > broadcast based > > > > >>>>>>> mco query but when I run direct addressing based mco query > then > > > the > > > > >>>>>>> response message is flowing everywhere instead of going to > > > consumer server > > > > >>>>>>> only. I have configured the exclude destination on Broker-A > to > > > make sure > > > > >>>>>>> none of the brokers are connected with the reply-to queue > name > > > except the > > > > >>>>>>> consumer server broker. > > > > >>>>>>> > > > > >>>>>>> On Sunday, January 28, 2018 at 2:08:06 AM UTC+5:30, > R.I.Pienaar > > > > >>>>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> On Sat, 27 Jan 2018, at 15:03, kk21987 wrote: > > > > >>>>>>>> > Hi, > > > > >>>>>>>> > > > > > >>>>>>>> > Am having some issue with --reply-to behavior and not > sure if > > > its > > > > >>>>>>>> an > > > > >>>>>>>> > issue > > > > >>>>>>>> > or the behavior itself like that. My setup is drawn in > > > > >>>>>>>> > ...... > > > > >>>>>>>> > > > > > >>>>>>>> > > > > > >>>>>>>> > My environment is enabled with direct_addressing. > Basically I > > > > >>>>>>>> always fire > > > > >>>>>>>> > the mco and forget it and don't want to consume any > messages > > > on > > > > >>>>>>>> mco client. > > > > >>>>>>>> > I have configured separate consumer server where it will > > > consume > > > > >>>>>>>> the > > > > >>>>>>>> > messages on the queue whatever declared during the mco > query > > > and > > > > >>>>>>>> will get > > > > >>>>>>>> > result from thie consumer server. The reason for > configured > > > > >>>>>>>> separate > > > > >>>>>>>> > consumer server is some commands whatever I run will take > > > more > > > > >>>>>>>> time to > > > > >>>>>>>> > execute and also doing some logic on the response > messages > > > based > > > > >>>>>>>> on my > > > > >>>>>>>> > requirement. > > > > >>>>>>>> > > > > > >>>>>>>> > I have configured exclude destination to make sure the > queue > > > > >>>>>>>> whatever am > > > > >>>>>>>> > using in reply-to not connected with Broker-A from other > > > brokers. > > > > >>>>>>>> > > > > > >>>>>>>> > When I run mco query using regex filter it pubish message > in > > > > >>>>>>>> broadcast mode > > > > >>>>>>>> > and consumer server consume the response without any > issue. > > > At > > > > >>>>>>>> the same > > > > >>>>>>>> > time when I check on the Broker-A ActiveMQ UI monitoring > I > > > > >>>>>>>> noticed Messages > > > > >>>>>>>> > Enqueued Dequeued count is NOT increased which means > nothing > > > > >>>>>>>> coming back to > > > > >>>>>>>> > Broker-A. > > > > >>>>>>>> > > > > > >>>>>>>> > But when I run mco query using direct addressing mode > (-I) > > > along > > > > >>>>>>>> with > > > > >>>>>>>> > reply-to the message is published as direct addressing > and > > > > >>>>>>>> consumer server > > > > >>>>>>>> > consume the response fine here as well. At the same time > when > > > I > > > > >>>>>>>> check on > > > > >>>>>>>> > the Broker-A ActiveMQ UI monitoring I noticed Messages > > > Enqueued > > > > >>>>>>>> Dequeued > > > > >>>>>>>> > count is increased which means message is coming to > Broker-A > > > as > > > > >>>>>>>> well at the > > > > >>>>>>>> > same time message is available on consumer server. > > > > >>>>>>>> > > > > > >>>>>>>> > Am not sure such difference between direct addressing and > > > > >>>>>>>> broadcast even > > > > >>>>>>>> > though both use the same reply-to mechanism. I don't want > any > > > > >>>>>>>> response > > > > >>>>>>>> > messages come back to Broker-A and all the response > messages > > > > >>>>>>>> should go to > > > > >>>>>>>> > Consumer Server only. Can I have some lights on this > please? > > > > >>>>>>>> > > > > >>>>>>>> what you describe should work, I guess the best hints lie > in > > > the > > > > >>>>>>>> server debug log. Does it send a reply and does it do so > to > > > the right > > > > >>>>>>>> target? > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> -- > > > > >>>>>>>> R.I.Pienaar / www.devco.net / @ripienaar > > > > >>>>>>>> > > > > >>>>>>> -- > > > > >>>>>> > > > > >>>>>> --- > > > > >>>>>> You received this message because you are subscribed to the > > > Google > > > > >>>>>> Groups "mcollective-users" group. > > > > >>>>>> To unsubscribe from this group and stop receiving emails from > it, > > > > >>>>>> send an email to mcollective-us...@googlegroups.com. > > > > >>>>>> For more options, visit https://groups.google.com/d/optout. > > > > >>>>>> > > > > >>>>>> -- > > > > >>>> > > > > >>>> --- > > > > >>>> You received this message because you are subscribed to the > Google > > > > >>>> Groups "mcollective-users" group. > > > > >>>> To unsubscribe from this group and stop receiving emails from > it, > > > send > > > > >>>> an email to mcollective-us...@googlegroups.com. > > > > >>>> For more options, visit https://groups.google.com/d/optout. > > > > >>>> > > > > >>>> -- > > > > >>> > > > > >>> --- > > > > >>> You received this message because you are subscribed to the > Google > > > > >>> Groups "mcollective-users" group. > > > > >>> To unsubscribe from this group and stop receiving emails from > it, > > > send > > > > >>> an email to mcollective-us...@googlegroups.com. > > > > >>> For more options, visit https://groups.google.com/d/optout. > > > > >>> > > > > >>> -- > > > > >> > > > > >> --- > > > > >> You received this message because you are subscribed to the > Google > > > Groups > > > > >> "mcollective-users" group. > > > > >> To unsubscribe from this group and stop receiving emails from it, > > > send an > > > > >> email to mcollective-us...@googlegroups.com. > > > > >> For more options, visit https://groups.google.com/d/optout. > > > > >> > > > > >> -- > > > > > > > > > > --- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "mcollective-users" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > send > > > an > > > > > email to mcollective-us...@googlegroups.com <javascript:>. > > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > > > > > -- > > > > > > > > --- > > > > You received this message because you are subscribed to the Google > > > > Groups "mcollective-users" group. > > > > To unsubscribe from this group and stop receiving emails from it, > send > > > > an email to mcollective-us...@googlegroups.com <javascript:>. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > -- > > > R.I.Pienaar / www.devco.net / @ripienaar > > > > > > > -- > > > > --- > > You received this message because you are subscribed to the Google > > Groups "mcollective-users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to mcollective-us...@googlegroups.com <javascript:>. > > For more options, visit https://groups.google.com/d/optout. > > > -- > R.I.Pienaar / www.devco.net / @ripienaar > -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to mcollective-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
