Check and insert
On 10/12/15, Ankur Garg <[email protected]> wrote: > But what if MongoDb bolt has some error , in that case I suppose the entire > tuple will be replayed from Spout meaning it will have to redo the > operation of inserting into sql . Is there a way I can skip inserting into > mysql ? > > On Mon, Oct 12, 2015 at 1:54 PM, Susheel Kumar Gadalay > <[email protected]> > wrote: > >> It is better to have 2 bolts - mysql bolt and mongodb bolt. >> >> Let mysql bolt forward the tuple to mongodb bolt, so in case of error >> it won't emit. >> >> On 10/12/15, Ankur Garg <[email protected]> wrote: >> > So I have a situation where the tuple received on Spout has to be saved >> to >> > mysql database and mongoDb as well . >> > >> > What should be better . Using 1 bolt to save it into mysql and MongoDb >> or 2 >> > seperate Bolts (One for saving into mysql and other for saving into >> Mongo). >> > >> > What happens when the exception occurs while saving into mysql ? I >> believe >> > I will get acknowledgement inside the fail method in my Spout . So If I >> > reprocess it using 2 bolts , I believe it will again be sent to Bolt >> > for >> > saving into Mongo database . >> > >> > If the above is true , will having 2 seperate bolts be of any advantage >> > ? >> > how can I configure things so that Failure while inserting into mysql >> does >> > not impact inserting into MongoDb . >> > >> > Thanks >> > Ankur >> > >> > On Sun, Oct 11, 2015 at 10:57 PM, Ravi Sharma <[email protected]> >> wrote: >> > >> >> That depends if ur spout error has affected jvm or normal application >> >> error >> >> >> >> performance issue in case of lot of errors, I don't think there is any >> >> issue be coz of errors themselves but ofcourse if u r retrying these >> >> messages on failure then that means u will be processing lot of >> >> messages >> >> then normal and overall throughput will go down >> >> >> >> Ravi >> >> >> >> If ur topology has enabled acknowledgment that means spout will always >> >> receive >> >> On 11 Oct 2015 18:15, "Ankur Garg" <[email protected]> wrote: >> >> >> >>> >> >>> Thanks for the reply Abhishek and Ravi . >> >>> >> >>> One question though , going with One topology with multiple spouts >> >>> ...What if something goes wrong in One spout or its associated bolts >> >>> .. >> >>> Does it impact other Spout as well? >> >>> >> >>> Thanks >> >>> Ankur >> >>> >> >>> On Sun, Oct 11, 2015 at 10:21 PM, Ravi Sharma <[email protected]> >> >>> wrote: >> >>> >> >>>> No 100% right ansers , u will have to test and see what will fit.. >> >>>> >> >>>> persoanlly i wud suggest Multiple spouts in one Topology and if you >> >>>> have >> >>>> N node where topology will be running then each Spout(reading from >> >>>> one >> >>>> queue) shud run N times in parallel. >> >>>> >> >>>> if 2 Queues and say 4 Nodes >> >>>> then one topolgy >> >>>> 4 Spouts reading from Queue1 in different nodes >> >>>> 4 spouts reading from Queue2 in different nodes >> >>>> >> >>>> Ravi. >> >>>> >> >>>> On Sun, Oct 11, 2015 at 5:25 PM, Abhishek priya < >> >>>> [email protected]> wrote: >> >>>> >> >>>>> I guess this is a question where there r no really correct answers. >> >>>>> I'll certainly avoid#1 as it is better to keep logic separate and >> >>>>> lightweight. >> >>>>> >> >>>>> If your downstream bolts are same, then it makes senses to keep >> >>>>> them >> >>>>> in >> >>>>> same topology but if they r totally different, I'll keep them in >> >>>>> two >> >>>>> different topologies. That will allow me to independently deploy >> >>>>> and >> >>>>> scale >> >>>>> the topology. But if the rest of logic is same I topology scaling >> >>>>> and >> >>>>> resource utilization will be better with one topology. >> >>>>> >> >>>>> I hope this helps.. >> >>>>> >> >>>>> Sent somehow.... >> >>>>> >> >>>>> > On Oct 11, 2015, at 9:07 AM, Ankur Garg <[email protected]> >> >>>>> > wrote: >> >>>>> > >> >>>>> > Hi , >> >>>>> > >> >>>>> > So I have a situation where I want to read messages from >> >>>>> > different >> >>>>> queues hosted in a Rabbitmq Server . >> >>>>> > >> >>>>> > Now , there are three ways which I can think to leverage Apache >> >>>>> > Storm >> >>>>> here :- >> >>>>> > >> >>>>> > 1) Use the same Spout (say Spout A) to read messages from >> >>>>> > different >> >>>>> queues and based on the messages received emit it to different >> >>>>> Bolts. >> >>>>> > >> >>>>> > 2) Use different Spout (Spout A and Spout B and so on) within the >> >>>>> same topology (say Topology A) to read messages from different >> >>>>> queues >> >>>>> . >> >>>>> > >> >>>>> > 3) Use Different Spouts one within eachTopology (Topology A , >> >>>>> Topology B and so on) to read messages from different queues . >> >>>>> > >> >>>>> > Which is the best way to process this considering I want high >> >>>>> throughput (more no of queue messages to be processed concurrently) >> >>>>> . >> >>>>> > >> >>>>> > Also , If In use same Topology for all Spouts (currently though >> >>>>> requirement is for 2 spouts) will failure in one Spout (or its >> >>>>> associated >> >>>>> Bolts) effect the second or will they both continue working >> separately >> >>>>> even >> >>>>> if some failure is in Spout B ? >> >>>>> > >> >>>>> > Cost wise , how much would it be to maintain two different >> >>>>> > topologies >> >>>>> . >> >>>>> > >> >>>>> > Looking for inputs from members here. >> >>>>> > >> >>>>> > Thanks >> >>>>> > Ankur >> >>>>> > >> >>>>> > >> >>>>> >> >>>> >> >>>> >> >>> >> > >> >
