Promod, You are right about applying the value in same window. And yes I don't want to change number of partitions or even redeploy any partition for this. I need to explore on Chandini's suggestion of using StatsListener.
-Priyanka On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <[email protected]> wrote: > I am not sure about priyanka's requirements but typically you would want to > apply the value in the same window for all partitions for consistency with > the calculations. This could be done two ways using a separate logical > operator feeding the value to all partitions so that the value appears in > the same window or repartitioning the existing set and setting the value > during repartitioning. In the latter case you are just repartitioning so > that the partitions start from same point with the new value and not to > change the number of partitions. > > Thanks > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <[email protected]> > wrote: > > > Pramod, > > > > Tim's discussion was about setting a property on multiple operators > > (different logical operators). > > Priyanka's question is about different partitions of an operator > > > > Let's say they are 5 partitions of an operator and we need to change an > int > > property. There are following ways: > > > > 1. Divide the int property by 5 and issue set-physical operator property > > command for each partition. This is manual. > > > > 2. Set the logical operator property but wait till partition happens. > This > > will require partitioning to be triggered for which a StatsListener needs > > to be implemented. So if partitioning never happens this will never be > > triggered. Also un-deploying and deploying a partition just to change a > > property on the physical partition seems un-necessary > > > > 3. Implementing StatsListener to send operator request and not > re-partition > > seems to be appropriate way. > > > > Chandni > > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni < > [email protected]> > > wrote: > > > > > Priyanka you want to divide the value among existing partitions right > and > > > not necessarily change the number of partitions. To this end Ashwin is > > > suggesting makes sense. However different partitions can be at > different > > > windows and will see the value change in different windows. You need a > > > separate operator to receive this change and propagate it to the > > partitions > > > via data flow. Tim discussed this over the last weekend in a series of > > > emails. > > > > > > Thanks > > > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale < > > > [email protected]> > > > wrote: > > > > > > > Thanks for suggestions. > > > > > > > > Ashwin, The option you proposed is doable, but then each operator > which > > > > needs to address such use case will have to track the number of > > > partitions. > > > > So we were looking for some common mechanism. > > > > Chandni, Let me try the res.operatorRequests, that could solve the > > > problem. > > > > > > > > -Priyanka > > > > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh < > > [email protected]> > > > > wrote: > > > > > > > > > Hi Priyanka, > > > > > > > > > > As Isha mentioned you will can implement a StatsListener. However > > > instead > > > > > of setting res.repartitionRequired = true, you can add a > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This will > not > > > > > re-deploy the instances. > > > > > > > > > > The OperatorRequest that you will need to implement will have to > > change > > > > the > > > > > property value. > > > > > > > > > > You can refer to AutoMetricTest for an example. > > > > > > > > > > - Chandni > > > > > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta < > > > > > [email protected]> wrote: > > > > > > > > > > > Why not maintain a variable containing the number of partitions > in > > > each > > > > > > partition; and when you set the property, just divide it by > number > > of > > > > > > partitions variable? Does it not solve the problem? > > > > > > > > > > > > Regards, > > > > > > Ashwin. > > > > > > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale < > > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I have an integer property named "intProp". This property is > > > > applicable > > > > > > to > > > > > > > an operator A. I can set the property from dtcli using > > > > > > > "set-operator-property operator-name property-name > > property-value" > > > > and > > > > > > this > > > > > > > will get applied to all instances of opeartor A. > > > > > > > > > > > > > > Now, I want to apply this property in a way that it's equally > > > divided > > > > > > > amongst all instances of operator A. For example, if > intProp=100 > > > and > > > > > > > number of partitions=4 then intProp=25 should be set on each > > > instance > > > > > of > > > > > > > operator A. > > > > > > > > > > > > > > Is there a way to achieve this? > > > > > > > > > > > > > > I have tried following approach to do this: I have overriden > > > > > > > definePartitions of Operator, once we do partitioning I am > > setting > > > > > right > > > > > > > value on each operator instance based on new partition count. > The > > > > > > > limitation here is that the value is applied during > > > definePartitions. > > > > > If > > > > > > > the value is updated dynamically it won't be applied till > > > > > > definePartitions > > > > > > > is called again. > > > > > > > > > > > > > > -Priyanka > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Regards, > > > > > > Ashwin. > > > > > > > > > > > > > > > > > > > > >
