> On Oct. 1, 2013, 4:35 p.m., Sriram Subramanian wrote: > > One other thing - > > > > 1. We have a bunch of features that we want to implement using the chooser. > > > > a. bootstrapping > > b. priority > > c. batching > > d. throttle > > > > Leaving the features above, the chooser is also used to do what it is meant > > for, to choose between equal stream partitions based on some heuristic > > (Example, Round robin). > > > > The way the features above should be specified in the config should have no > > relation to the chooser. For example, > > > > task.bootstrap = stream1,stream2 > > > > task.prioriites.stream1 = 2 > > task.priorities.stream2 = 3 > > > > task.batch.size = 100 > > > > task.throttle.stream1 = 10 > > > > task.message.chooser = RoundRobin > > > > All the above features should work irrespective of the chooser specified. > > Currently, we are using a DefaultChooser that composes all the choosers and > > orchestrates them. The problem is that the user of the framework will have > > no idea what the default chooser does. If they replace it with say > > RoundRobin, all the features above will break. So, instead of using a > > wrapped.chooser, we should just use the DefaultChooser/ChooserOrchestrator > > internally but never expose that. The task.message.chooser would determine > > the default behavior and the rest of the features will just co-exist with > > whatever chooser that is chosen. > >
Agree. I was thinking this as well. I'm going to move all of the wiring in DefaultChooserFactory for batch/bootstrap/priority into the SamzaContainer, instead. I'll also move the DefaultChooserConfig stuff into TaskConfig. This will also take care of eliminating task.chooser.wrapped.class (1), as we can just use task.chooser.class instead. - Chris ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13725/#review26564 ----------------------------------------------------------- On Sept. 26, 2013, 3:03 a.m., Chris Riccomini wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/13725/ > ----------------------------------------------------------- > > (Updated Sept. 26, 2013, 3:03 a.m.) > > > Review request for samza. > > > Repository: samza > > > Description > ------- > > wrote default chooser test that bootstraps, prioritizes, and batches. > > > turning on priority tests for default chooser. refactor default chooser a bit > to make it easier to test. > > > test default chooser with just round robin. > > > bug fix to use round robin chooser factory in default chooser > > > change wiring for default chooser to make it more testable. add > start/stop/register tests to all choosers. > > > add one more test in priority chooser. > > > add unit test for tiered priority chooser > > > add unit test for bootstrapping chooser. > > > turn default chooser on in samza container. add license to test stateful > task. add more docs to default chooser. clean up batching unit test. > > > add unit test for batching chooser. > > > only wire in wrapping message choosers when we need them. add docs to the > default chooser factory. > > > rename class to BootstrappingChooser. > > > refactor to move into org.apache.samza.system.chooser. > > > build latest message offset map. > > > add a streams-behind-chooser that guarantees one message from each > SystemStream before choose is called. > > > add start/stop/register back. all tests pass. > > > fix bug -- should allow manual override if Int.MaxInt for bootstrap streams. > > > minor bug in default chooser. was re-using same chooser everywhere. > > > adding wiring in default chooser. > > > initial pass adding composed message choosers. > > > rebase to master, which includes SAMZA-25 metrics. fix several tests that > were broken after removing start/stop/register. > > > adding more docs for round robin > > > remove start/stop/register > > > cleanup some wiki markdown in MessageChooser javadoc. > > > add more javadocs to the message chooser. > > > Merge branch 'SAMZA-2_fine-grain-control-over-stream-consumption' of > github.com:criccomini/incubator-samza into > SAMZA-2_fine-grain-control-over-stream-consumption > > > added start, stop, and register to message chooser. > > > adding docs for message chooser. swiching round robin chooser back to a queue. > > > missed license in message chooser factory > > > add apache licensing > > > samza container was using message chooser, not message chooser factory. fixed. > > > add stream chooser test. update stream chooser to invert priority due to bug. > > > add round robin test. fix compile error in round robin chooser. > > > add priority chooser test. fix bug in priority chooser that was reversing > ordering. > > > adding stream chooser. adding message chooser factory. > > > adding priority chooser. moving default chooser to round robin chooser. > adding config for chooser > > > Diffs > ----- > > docs/learn/documentation/0.7.0/container/streams.md > b15c34d3f68e32d56e0da8af91d78e47a5110f67 > samza-api/src/main/java/org/apache/samza/system/MessageChooser.java > 306b2902303c72f3d7a3eb313f55d7e88d21e00d > samza-api/src/main/java/org/apache/samza/system/SystemAdmin.java > c902d414484e05ae75c9ca58ad9629cb01120f62 > samza-api/src/main/java/org/apache/samza/system/chooser/MessageChooser.java > PRE-CREATION > > samza-api/src/main/java/org/apache/samza/system/chooser/PriorityChooser.java > PRE-CREATION > > samza-api/src/main/java/org/apache/samza/util/SinglePartitionSystemAdmin.java > e4ed30bf363142d82a3c40909e160b5825fe60fd > > samza-api/src/test/java/org/apache/samza/system/chooser/TestPriorityChooser.java > PRE-CREATION > > samza-core/src/main/scala/org/apache/samza/config/DefaultChooserConfig.scala > PRE-CREATION > samza-core/src/main/scala/org/apache/samza/config/TaskConfig.scala > 0c742d83c2f60d2448a79376677713a1ff0b11ec > samza-core/src/main/scala/org/apache/samza/container/SamzaContainer.scala > 62bd243115e71612e00784124baa972b33e56cb7 > samza-core/src/main/scala/org/apache/samza/system/DefaultChooser.scala > 5a72e7a3bfba0f06a5a98c6ba26865800d7780b9 > samza-core/src/main/scala/org/apache/samza/system/SystemConsumers.scala > d24671ec64a42ede6f779effe9c845e1cbbc5e51 > > samza-core/src/main/scala/org/apache/samza/system/chooser/BatchingChooser.scala > PRE-CREATION > > samza-core/src/main/scala/org/apache/samza/system/chooser/BootstrappingChooser.scala > PRE-CREATION > > samza-core/src/main/scala/org/apache/samza/system/chooser/DefaultChooser.scala > PRE-CREATION > > samza-core/src/main/scala/org/apache/samza/system/chooser/RoundRobinChooser.scala > PRE-CREATION > > samza-core/src/main/scala/org/apache/samza/system/chooser/TieredPriorityChooser.scala > PRE-CREATION > samza-core/src/test/scala/org/apache/samza/system/TestSystemConsumers.scala > PRE-CREATION > > samza-core/src/test/scala/org/apache/samza/system/chooser/MockMessageChooser.scala > PRE-CREATION > > samza-core/src/test/scala/org/apache/samza/system/chooser/TestBatchingChooser.scala > PRE-CREATION > > samza-core/src/test/scala/org/apache/samza/system/chooser/TestBootstrappingChooser.scala > PRE-CREATION > > samza-core/src/test/scala/org/apache/samza/system/chooser/TestDefaultChooser.scala > PRE-CREATION > > samza-core/src/test/scala/org/apache/samza/system/chooser/TestRoundRobinChooser.scala > PRE-CREATION > > samza-core/src/test/scala/org/apache/samza/system/chooser/TestTieredPriorityChooser.scala > PRE-CREATION > > samza-kafka/src/main/scala/org/apache/samza/system/kafka/KafkaSystemAdmin.scala > 183c6ccce39dedaef9dba56d5b61ffdedfc9d08a > > samza-test/src/test/scala/org/apache/samza/test/integration/TestStatefulTask.scala > 7d4e20a8bdc7a45b0a1b464a6f4b868d1d03eab0 > > samza-yarn/src/test/scala/org/apache/samza/job/yarn/TestSamzaAppMasterTaskManager.scala > 68050528cd5f8acfe3a1f7563b4e7fe6c7473be5 > > Diff: https://reviews.apache.org/r/13725/diff/ > > > Testing > ------- > > > Thanks, > > Chris Riccomini > >
