I am not sure about priyanka's requirements but typically you would want to apply the value in the same window for all partitions for consistency with the calculations. This could be done two ways using a separate logical operator feeding the value to all partitions so that the value appears in the same window or repartitioning the existing set and setting the value during repartitioning. In the latter case you are just repartitioning so that the partitions start from same point with the new value and not to change the number of partitions.
Thanks On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <[email protected]> wrote: > Pramod, > > Tim's discussion was about setting a property on multiple operators > (different logical operators). > Priyanka's question is about different partitions of an operator > > Let's say they are 5 partitions of an operator and we need to change an int > property. There are following ways: > > 1. Divide the int property by 5 and issue set-physical operator property > command for each partition. This is manual. > > 2. Set the logical operator property but wait till partition happens. This > will require partitioning to be triggered for which a StatsListener needs > to be implemented. So if partitioning never happens this will never be > triggered. Also un-deploying and deploying a partition just to change a > property on the physical partition seems un-necessary > > 3. Implementing StatsListener to send operator request and not re-partition > seems to be appropriate way. > > Chandni > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <[email protected]> > wrote: > > > Priyanka you want to divide the value among existing partitions right and > > not necessarily change the number of partitions. To this end Ashwin is > > suggesting makes sense. However different partitions can be at different > > windows and will see the value change in different windows. You need a > > separate operator to receive this change and propagate it to the > partitions > > via data flow. Tim discussed this over the last weekend in a series of > > emails. > > > > Thanks > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale < > > [email protected]> > > wrote: > > > > > Thanks for suggestions. > > > > > > Ashwin, The option you proposed is doable, but then each operator which > > > needs to address such use case will have to track the number of > > partitions. > > > So we were looking for some common mechanism. > > > Chandni, Let me try the res.operatorRequests, that could solve the > > problem. > > > > > > -Priyanka > > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh < > [email protected]> > > > wrote: > > > > > > > Hi Priyanka, > > > > > > > > As Isha mentioned you will can implement a StatsListener. However > > instead > > > > of setting res.repartitionRequired = true, you can add a > > > > StatsListener.OperatorRequest to res.operatorsRequests. This will not > > > > re-deploy the instances. > > > > > > > > The OperatorRequest that you will need to implement will have to > change > > > the > > > > property value. > > > > > > > > You can refer to AutoMetricTest for an example. > > > > > > > > - Chandni > > > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta < > > > > [email protected]> wrote: > > > > > > > > > Why not maintain a variable containing the number of partitions in > > each > > > > > partition; and when you set the property, just divide it by number > of > > > > > partitions variable? Does it not solve the problem? > > > > > > > > > > Regards, > > > > > Ashwin. > > > > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > I have an integer property named "intProp". This property is > > > applicable > > > > > to > > > > > > an operator A. I can set the property from dtcli using > > > > > > "set-operator-property operator-name property-name > property-value" > > > and > > > > > this > > > > > > will get applied to all instances of opeartor A. > > > > > > > > > > > > Now, I want to apply this property in a way that it's equally > > divided > > > > > > amongst all instances of operator A. For example, if intProp=100 > > and > > > > > > number of partitions=4 then intProp=25 should be set on each > > instance > > > > of > > > > > > operator A. > > > > > > > > > > > > Is there a way to achieve this? > > > > > > > > > > > > I have tried following approach to do this: I have overriden > > > > > > definePartitions of Operator, once we do partitioning I am > setting > > > > right > > > > > > value on each operator instance based on new partition count. The > > > > > > limitation here is that the value is applied during > > definePartitions. > > > > If > > > > > > the value is updated dynamically it won't be applied till > > > > > definePartitions > > > > > > is called again. > > > > > > > > > > > > -Priyanka > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > Ashwin. > > > > > > > > > > > > > > >
