There is a discussion about stream partitioning v/s PE partitioning in that JIRA. I had some implementation that allows both types of partitioning schemes.
https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=blob;f=subprojects/s4-tools/src/main/java/org/apache/s4/tools/helix/CreateTask.java;h=36a45bc5bef5e4c4f56bfb4c740abb315197ca21;hb=59bcb94cc727cb1600074e6da81330953aeaa566 When you add the task, you can use pe type s the name of the task and specify the streams it is interested in. Give it a try :-) On Wed, Dec 11, 2013 at 8:50 AM, Sergio Vavassori <[email protected]>wrote: > > > > It's my understanding that a cluster is a group of nodes and each node > has > >> the same application-code copy; this means that if I want to partition > the > >> ProcessingElements between nodes I need to group them in different > >> clusters. > >> So, mapping S4 elements into a "classical" Stream Processing naming > >> (Nodes, > >> Operators, Slides...), would be having one application (Operator) per > >> cluster and configure the ProcessingElements as singleton (1 Slide per > >> Node). > >> > > > > you should use a key to partition your stream. Use the KeyFinder to > > identify keys in events. > > > but this is Stream Partitioning, not Processing Element Partitioning. I > could use this system, but for my needs I should then multiply each message > for the number of partitions and I still wouldn’t be able to control PE > allocation between types in the same nodes. > > E.g. if I have: > > * 2 PE types: PE1 and PE2, configured as singletons. > * 3 nodes > > in each node both PE types can be allocated, so 6 instead of 3. Is there a > way to allocate only one PE per node? > > Regards, > Sergio Vavassori >
