here's a batch fire and forget client, of course it doesnt return instantly but it doesnt expect replies or wait for replies
https://gist.github.com/ripienaar/52e73db531cc22b434b8433ababd8df2 On Sun, 28 Jan 2018, at 16:51, kk21987 wrote: > It would be good if I get true batch mode where all requests have the same > ID, since my consumer segregates the response output based on the ID only. > > The reason why I want to go with batch mode is, I have some set of servers > where I need to collect some data. But in this the data and also the > servers list will keep vary which comes from different subcollectives. So I > don't have specific facts filter to run in broadcast mode and also I can't > make such facts also. Its basically kind of requirement for us to run this > often. > > > On Sunday, January 28, 2018 at 9:14:52 PM UTC+5:30, R.I.Pienaar wrote: > > > > > > > On Sun, 28 Jan 2018, at 15:29, kk21987 wrote: > > > Yes please. If the reply-to works well with batch mode it would be great > > > for me as it will solve my proble :) > > > > yeah fire and forget doesnt support batch mode in the normal client > > > > in your use case does it matter if the replies all have the same request > > id - ie. do you want a true batch mode where all requests have the same ID > > or many seperate requests would be fine? > > > > I still am curious why you need to use direct mode rather than broadcast > > > > > > On Sunday, January 28, 2018 at 6:23:35 PM UTC+5:30, R.I.Pienaar wrote: > > > > > > > > > > > > On 28 Jan 2018, at 13:31, kk21987 <kkarth...@gmail.com <javascript:>> > > > > wrote: > > > > > > > > Yeah I agree it doesn't make sense to run without batch in direct > > address > > > > mode. But there is one problem using batch with reply-to. > > > > > > > > I have set the export for MCOLLECTIVE_EXTRA_OPTS with the values batch > > > > size as 200 and batch sleep as 30 and when I run mco query with > > --reply-to > > > > its returning the result on the console of mco client server itself > > which I > > > > don't want. Since I have separate consumer server configured for this. > > > > > > > > I really don't understand why it behaves like this. Any idea please. > > > > > > > > > > > > Could be batch mode just isn’t supported in the library when using > > reply > > > > to. I can look later when at a computer again > > > > > > > > Would be trivial to write a small custom application that does this > > > > though. > > > > > > > > > > > > On Sunday, January 28, 2018 at 3:46:04 PM UTC+5:30, R.I.Pienaar wrote: > > > > > > > >> > > > >> > > > >> On 28 Jan 2018, at 11:13, kk21987 <kkarth...@gmail.com> wrote: > > > >> > > > >> Understood and thanks for the detailed clarification! I have already > > > >> splitted as different sub collectives (2000 per subcollective) but > > again > > > >> all the subcollectives are connected with Broker-A which will act as > > > >> Central(just for naming convention) and this central will get big > > pressure > > > >> when I run direct addressing against huge number of servers. So is > > there > > > >> any feasibility to connect all subcollectives ActiveMQ brokers and > > run the > > > >> command instead of using Central one by declaring some parameter in > > > >> mcollective client config or using any other tool in between? I think > > which > > > >> this way it will go smooth, since I see mcollective publish the > > message > > > >> very quickly to Broker-A(central) and as you said this is getting big > > > >> pressure. > > > >> > > > >> > > > >> Why do you need to use direct addressing for your use case? If your > > node > > > >> lists arbitrary and not something you can discover against? > > > >> > > > >> You don’t use any of the confirmation features it brings? > > > >> > > > >> Only other option at this scale might be to direct address + batches > > > >> > > > >> > > > >> On Sunday, January 28, 2018 at 3:35:01 PM UTC+5:30, R.I.Pienaar > > wrote: > > > >>> > > > >>> > > > >>> > > > >>> On 28 Jan 2018, at 10:48, kk21987 <kkarth...@gmail.com> wrote: > > > >>> > > > >>> I agree. I read about the Choria and very interested to use it. But > > am > > > >>> running little old version of Ruby, Puppet where I need to plan it > > for > > > >>> upgrade. But still Choria is in active dev am little concern to go > > ahead it > > > >>> on Production. > > > >>> > > > >>> Between I dig into furthermore and found its issue with direct > > > >>> addressing mode only. When I run mco running with -T mcollective or > > > >>> whatever subcollectives it goes very smooth, but when I run mco > > query using > > > >>> flatfile or -I(direct addressing) against high number of servers > > only then > > > >>> the issue occurs. > > > >>> > > > >>> So I suspect direct addressing takes time to execute and would > > curious > > > >>> to understand why it gives such difference. I believe direct > > addressing > > > >>> should be fast compare with broadcast, correct me if am wrong. > > > >>> > > > >>> > > > >>> It’s a trade off because direct has to create and distribute a > > single > > > >>> message per every target. So if you have 10k targets it’s that many > > > >>> messages. Big pressure on activemq especially with the single queue > > and > > > >>> processing the per node selectors > > > >>> > > > >>> Broadcast in turn is much lighter on the middleware and much faster > > all > > > >>> round but you don’t know which nodes will get it > > > >>> > > > >>> UDP cheap and fast because it’s not guaranteed > > > >>> TCP expensive and slow with many layers but guaranteed delivery (or > > at > > > >>> least advisory about non delivery ack) > > > >>> > > > >>> Likewise with mco broadcast and direct. > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> On Sunday, January 28, 2018 at 3:11:34 PM UTC+5:30, R.I.Pienaar > > wrote: > > > >>> > > > >>>> > > > >>>> > > > >>>> On 28 Jan 2018, at 10:35, kk21987 <kkarth...@gmail.com> wrote: > > > >>>> > > > >>>> Some improvement with your debug help! Looks like the reply-to is > > > >>>> working perfect as expected. The actual problem is triggering the > > command > > > >>>> to all the servers. > > > >>>> > > > >>>> I have total of 9824 servers connected with mcollective.nodes and > > when > > > >>>> I run mco query against all the servers to execute some command > > then > > > >>>> dispatching that mco query message all the consumers takes time > > which is > > > >>>> why the pending messages are there and whenever the each consumers > > receives > > > >>>> the message its slowly decreasing. > > > >>>> > > > >>>> > > > >>>> Great > > > >>>> > > > >>>> > > > >>>> > > > >>>> Now I need to look for some tuning to get the message dispatched to > > all > > > >>>> the consumer very quickly, so it will be good. Any suggestion or > > best > > > >>>> tuning parameters available for this. I will try to look for > > ActiveMQ > > > >>>> documentation to get the something. > > > >>>> > > > >>>> > > > >>>> No I gave up making activemq scale this far. I have some large > > rabbitmq > > > >>>> clusters but that requires a ton of hardware too > > > >>>> > > > >>>> Choria with NATS or the Choria broker would do this comfortably on > > a > > > >>>> single small 4GB VM with a tremendous speed improvement. > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Sunday, January 28, 2018 at 2:03:34 PM UTC+5:30, kk21987 wrote: > > > >>>> > > > >>>>> Thanks. > > > >>>>> > > > >>>>> Yes I captured the client debug log as well and could see that it > > > >>>>> publish message with reply-to properly. > > > >>>>> > > > >>>>> Direct Addressing: > > > >>>>> > > > >>>>> D, [2018-01-28T02:24:56.851292 #17287] DEBUG -- : > > activemq.rb:402:in > > > >>>>> `publish' Sending a direct message to ActiveMQ target > > > >>>>> '/queue/mcollective.nodes' with headers > > '{"mc_identity"=>"mco-server-name", > > > >>>>> "timestamp"=>"1517127896000", "expires"=>"1517127966000", > > > >>>>> "reply-to"=>"/queue/data.consumer.all"}' > > > >>>>> > > > >>>>> Broadcast: > > > >>>>> > > > >>>>> D, [2018-01-28T02:25:19.345299 #17340] DEBUG -- : > > activemq.rb:409:in > > > >>>>> `publish' Sending a broadcast message to ActiveMQ target > > > >>>>> '/topic/mcollective.shell.agent' with headers > > '{"expires"=>"1517127989000", > > > >>>>> "reply-to"=>"/queue/data.consumer.all", > > "timestamp"=>"1517127919000"}' > > > >>>>> > > > >>>>> Here is the way I used to exclude the queue consumer destination > > on > > > >>>>> Broker-A. > > > >>>>> > > > >>>>> <queue physicalName="data.consumer.>"/> > > > >>>>> > > > >>>>> I couldn't take the debug logs on ActiveMQ due to the servers > > count is > > > >>>>> little high, so it generate huge logs and couldn't get it exactly. > > But I > > > >>>>> used ActiveMQ-Cli to export the JMS queue and here is a example of > > one > > > >>>>> pending message > > > >>>>> > > > >>>>> > > > >>>>> <jms-message> > > > >>>>> <header> > > > >>>>> > > > >>>>> > > <message-id>ID:broker-A-16618-1517126209254-79:5:-1:1:559</message-id> > > > >>>>> <delivery-mode>1</delivery-mode> > > > >>>>> <destination>queue://mcollective.nodes</destination> > > > >>>>> <expiration>1517127457000</expiration> > > > >>>>> <priority>4</priority> > > > >>>>> <redelivered>false</redelivered> > > > >>>>> <reply-to>queue://data.consumer.all</reply-to> > > > >>>>> <timestamp>2018-01-28T02:16:27</timestamp> > > > >>>>> </header> > > > >>>>> <properties> > > > >>>>> <property> > > > >>>>> <name>content-type</name> > > > >>>>> <value>text/plain; charset=UTF-8</value> > > > >>>>> </property> > > > >>>>> <property> > > > >>>>> <name>mc_identity</name> > > > >>>>> <value>mcollective-server-name</value> > > > >>>>> </property> > > > >>>>> </properties> > > > >>>>> </jms-message> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> On Sunday, January 28, 2018 at 12:40:14 PM UTC+5:30, R.I.Pienaar > > wrote: > > > >>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> On 28 Jan 2018, at 06:06, kk21987 <kkarth...@gmail.com> wrote: > > > >>>>>> > > > >>>>>> My apologize. I misunderstood few things. > > > >>>>>> > > > >>>>>> Basically when I run mco query in direct addressing mode then it > > > >>>>>> publish that message as queue where as broadcast mode publish the > > message > > > >>>>>> as topic. So I suspect its not an issue with the modes whatever > > the mco > > > >>>>>> query use. The only thing I want to understand is, lets say I run > > mco query > > > >>>>>> against 1000servers with --reply-to (fire and forget, no > > acknowledgement > > > >>>>>> and nothing. Correct me if am wrong), so all the response message > > will go > > > >>>>>> to the consumer where the active subscriber is present for the > > given queue. > > > >>>>>> The response message is not coming to Broker-A which I have > > configured by > > > >>>>>> starting receiver here as well and perfectly its going to the > > consumer > > > >>>>>> where its active. > > > >>>>>> > > > >>>>>> > > > >>>>>> That sounds about right. Yes it will use a topic and all nodes > > will > > > >>>>>> get it > > > >>>>>> > > > >>>>>> > > > >>>>>> My concern is, at the same time I see count is showing in pending > > > >>>>>> message for the queue "mcollective.nodes" and slowly its > > decreasing(at the > > > >>>>>> same enqueued and dequeued messages count also increasing slowly) > > which I > > > >>>>>> have verified on ActiveMQ UI Jolokia monitoring (8161). So I > > would like to > > > >>>>>> understand why such messages count in pending messages, since all > > I want to > > > >>>>>> just fire the command and forget and don't want to worry about > > response or > > > >>>>>> whether its executed or not. > > > >>>>>> > > > >>>>>> Name ↑Number Of Pending Messages Number Of Consumers Messages > > > >>>>>> Enqueued Messages Dequeued Views Operations > > > >>>>>> mcollective.nodes 9 84 884 875 Browse Active Consumers > > > >>>>>> Active Producers > > > >>>>>> Send To PurgeDelete > > > >>>>>> > > > >>>>>> > > > >>>>>> > > > >>>>>> You will need to double check in the logs. Your client also logs > > and > > > >>>>>> at debug > > > >>>>>> level you can verify where the publish to > > > >>>>>> > > > >>>>>> > > > >>>>>> On Sunday, January 28, 2018 at 7:48:15 AM UTC+5:30, kk21987 > > wrote: > > > >>>>>> > > > >>>>>>> Based on my understanding when I run mco query with reply-to I > > > >>>>>>> should see the reply-to queue name on the response header and > > its there on > > > >>>>>>> mcollective server logs, which I have verified by enabling debug > > on the > > > >>>>>>> mcollective servers. > > > >>>>>>> > > > >>>>>>> My worry is, I shouldn't get any response messages back to the > > > >>>>>>> Broker-A and all the response messages should go to the consumer > > server > > > >>>>>>> which is running elsewhere. This is working as expected for > > broadcast based > > > >>>>>>> mco query but when I run direct addressing based mco query then > > the > > > >>>>>>> response message is flowing everywhere instead of going to > > consumer server > > > >>>>>>> only. I have configured the exclude destination on Broker-A to > > make sure > > > >>>>>>> none of the brokers are connected with the reply-to queue name > > except the > > > >>>>>>> consumer server broker. > > > >>>>>>> > > > >>>>>>> On Sunday, January 28, 2018 at 2:08:06 AM UTC+5:30, R.I.Pienaar > > > >>>>>>> wrote: > > > >>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> On Sat, 27 Jan 2018, at 15:03, kk21987 wrote: > > > >>>>>>>> > Hi, > > > >>>>>>>> > > > > >>>>>>>> > Am having some issue with --reply-to behavior and not sure if > > its > > > >>>>>>>> an > > > >>>>>>>> > issue > > > >>>>>>>> > or the behavior itself like that. My setup is drawn in > > > >>>>>>>> > ...... > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > My environment is enabled with direct_addressing. Basically I > > > >>>>>>>> always fire > > > >>>>>>>> > the mco and forget it and don't want to consume any messages > > on > > > >>>>>>>> mco client. > > > >>>>>>>> > I have configured separate consumer server where it will > > consume > > > >>>>>>>> the > > > >>>>>>>> > messages on the queue whatever declared during the mco query > > and > > > >>>>>>>> will get > > > >>>>>>>> > result from thie consumer server. The reason for configured > > > >>>>>>>> separate > > > >>>>>>>> > consumer server is some commands whatever I run will take > > more > > > >>>>>>>> time to > > > >>>>>>>> > execute and also doing some logic on the response messages > > based > > > >>>>>>>> on my > > > >>>>>>>> > requirement. > > > >>>>>>>> > > > > >>>>>>>> > I have configured exclude destination to make sure the queue > > > >>>>>>>> whatever am > > > >>>>>>>> > using in reply-to not connected with Broker-A from other > > brokers. > > > >>>>>>>> > > > > >>>>>>>> > When I run mco query using regex filter it pubish message in > > > >>>>>>>> broadcast mode > > > >>>>>>>> > and consumer server consume the response without any issue. > > At > > > >>>>>>>> the same > > > >>>>>>>> > time when I check on the Broker-A ActiveMQ UI monitoring I > > > >>>>>>>> noticed Messages > > > >>>>>>>> > Enqueued Dequeued count is NOT increased which means nothing > > > >>>>>>>> coming back to > > > >>>>>>>> > Broker-A. > > > >>>>>>>> > > > > >>>>>>>> > But when I run mco query using direct addressing mode (-I) > > along > > > >>>>>>>> with > > > >>>>>>>> > reply-to the message is published as direct addressing and > > > >>>>>>>> consumer server > > > >>>>>>>> > consume the response fine here as well. At the same time when > > I > > > >>>>>>>> check on > > > >>>>>>>> > the Broker-A ActiveMQ UI monitoring I noticed Messages > > Enqueued > > > >>>>>>>> Dequeued > > > >>>>>>>> > count is increased which means message is coming to Broker-A > > as > > > >>>>>>>> well at the > > > >>>>>>>> > same time message is available on consumer server. > > > >>>>>>>> > > > > >>>>>>>> > Am not sure such difference between direct addressing and > > > >>>>>>>> broadcast even > > > >>>>>>>> > though both use the same reply-to mechanism. I don't want any > > > >>>>>>>> response > > > >>>>>>>> > messages come back to Broker-A and all the response messages > > > >>>>>>>> should go to > > > >>>>>>>> > Consumer Server only. Can I have some lights on this please? > > > >>>>>>>> > > > >>>>>>>> what you describe should work, I guess the best hints lie in > > the > > > >>>>>>>> server debug log. Does it send a reply and does it do so to > > the right > > > >>>>>>>> target? > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> -- > > > >>>>>>>> R.I.Pienaar / www.devco.net / @ripienaar > > > >>>>>>>> > > > >>>>>>> -- > > > >>>>>> > > > >>>>>> --- > > > >>>>>> You received this message because you are subscribed to the > > Google > > > >>>>>> Groups "mcollective-users" group. > > > >>>>>> To unsubscribe from this group and stop receiving emails from it, > > > >>>>>> send an email to mcollective-us...@googlegroups.com. > > > >>>>>> For more options, visit https://groups.google.com/d/optout. > > > >>>>>> > > > >>>>>> -- > > > >>>> > > > >>>> --- > > > >>>> You received this message because you are subscribed to the Google > > > >>>> Groups "mcollective-users" group. > > > >>>> To unsubscribe from this group and stop receiving emails from it, > > send > > > >>>> an email to mcollective-us...@googlegroups.com. > > > >>>> For more options, visit https://groups.google.com/d/optout. > > > >>>> > > > >>>> -- > > > >>> > > > >>> --- > > > >>> You received this message because you are subscribed to the Google > > > >>> Groups "mcollective-users" group. > > > >>> To unsubscribe from this group and stop receiving emails from it, > > send > > > >>> an email to mcollective-us...@googlegroups.com. > > > >>> For more options, visit https://groups.google.com/d/optout. > > > >>> > > > >>> -- > > > >> > > > >> --- > > > >> You received this message because you are subscribed to the Google > > Groups > > > >> "mcollective-users" group. > > > >> To unsubscribe from this group and stop receiving emails from it, > > send an > > > >> email to mcollective-us...@googlegroups.com. > > > >> For more options, visit https://groups.google.com/d/optout. > > > >> > > > >> -- > > > > > > > > --- > > > > You received this message because you are subscribed to the Google > > Groups > > > > "mcollective-users" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > an > > > > email to mcollective-us...@googlegroups.com <javascript:>. > > > > For more options, visit https://groups.google.com/d/optout. > > > > > > > > > > > > > > -- > > > > > > --- > > > You received this message because you are subscribed to the Google > > > Groups "mcollective-users" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to mcollective-us...@googlegroups.com <javascript:>. > > > For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > R.I.Pienaar / www.devco.net / @ripienaar > > > > -- > > --- > You received this message because you are subscribed to the Google > Groups "mcollective-users" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to mcollective-users+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- R.I.Pienaar / www.devco.net / @ripienaar -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to mcollective-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
