Hi,
I've already changed the design of my code to solve the problem, but just
wanted to ask if the problem could have been solved alternatively.
*The problem:*
Wanted to create x number of spouts that would each handle a batch of
records from a database. First spout handles batch 1 to 1000. Second spout
handles 1001 to 2000 etc.
The problem is if x spouts are created dynamically, then how do we assign
the output of all the spouts to the bolts?
*Solution that didn't work:*
A colleague suggested
String s = "sp";
*for(*Integer ordinal = 1; ordinal <= *numSpouts*; ++ordinal) *{*
String spoutName = s + ordinal.toString();
builder.setSpout(spoutName, new mdSpout(), 1)
.setNumTasks(1);
*}*
*BoltDeclarer boltDec = builder.setBolt(cgBolt, new CBolt(), thr);*
*for(*int i=0; i < *numSpouts*; i++) *{*
*boltDec.*fieldsGrouping(s+Integer.toString(i), new
Fields(configRef.BID))
.allGrouping(s+Integer.toString(i), configRef.ID);
*}*
*But this gave an invalid topology exception.*
*Solution I eventually had to use:*
*for(*Integer ordinal = 1; ordinal <= numSpouts; ++ordinal) *{*
String spoutName = s + ordinal.toString();
builder.setSpout(spoutName, new mdSpout(), 1)
.setNumTasks(1);
*}*
*switch(numSpouts) {*
case 1:
builder.setBolt(cgBolt, new CBolt(), thr)
//.setNumTasks(3)
.fieldsGrouping(s+"1", new Fields(configRef.BID))
.allGrouping(s+"1", configRef.ID);
break;
case 2:
builder.setBolt(cgBolt, new CBolt(), thr)
.fieldsGrouping(s+"1", new Fields(configRef.BID))
.allGrouping(s+"1", ID)
.fieldsGrouping(s+"2", new Fields(configRef.BID))
.allGrouping(s+"2", configRef.ID);
break;
case 3:
builder.setBolt(cgBolt, new CBolt(), thr)
.fieldsGrouping(s+"1", new Fields(configRef.BID))
.allGrouping(s+"1", configRef.ID)
.fieldsGrouping(s+"2", new Fields(configRef.BID))
.allGrouping(s+"2", configRef.ID)
.fieldsGrouping(s+"3", new Fields(configRef.BID))
.allGrouping(s+"3", configRef.ID);
break;
*}*
This worked, but obviously there should be a better way to do this, right?
ps: Eventually I used a different method of sending the batches to the
spouts so that the spouts didn't have to be created with a for loop, but am
still curious about how to solve the above problem.
--
Regards,
Navin
