Pramod, What happens if the Separate Operator that is used to propagate this change is on say window 1000 and all the instances of Operator A are at 300..At this time user changes value of property. Then the new value will be delayed to Operator A instances by 700 windows
Thanks -Gaurav On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <[email protected]> wrote: > Pramod , > > That is what I was thinking that Operator requests can also be received in > different windows...Looks like adding another operator is safe solution > > Thanks > -Gaurav > > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <[email protected]> > wrote: > >> The operator request would also be received in different windows. >> >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale < >> [email protected]> >> wrote: >> >> > Promod, >> > >> > You are right about applying the value in same window. And yes I don't >> want >> > to change number of partitions or even redeploy any partition for this. >> > I need to explore on Chandini's suggestion of using StatsListener. >> > >> > -Priyanka >> > >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni < >> [email protected]> >> > wrote: >> > >> > > I am not sure about priyanka's requirements but typically you would >> want >> > to >> > > apply the value in the same window for all partitions for consistency >> > with >> > > the calculations. This could be done two ways using a separate logical >> > > operator feeding the value to all partitions so that the value >> appears in >> > > the same window or repartitioning the existing set and setting the >> value >> > > during repartitioning. In the latter case you are just repartitioning >> so >> > > that the partitions start from same point with the new value and not >> to >> > > change the number of partitions. >> > > >> > > Thanks >> > > >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh < >> [email protected] >> > > >> > > wrote: >> > > >> > > > Pramod, >> > > > >> > > > Tim's discussion was about setting a property on multiple operators >> > > > (different logical operators). >> > > > Priyanka's question is about different partitions of an operator >> > > > >> > > > Let's say they are 5 partitions of an operator and we need to >> change an >> > > int >> > > > property. There are following ways: >> > > > >> > > > 1. Divide the int property by 5 and issue set-physical operator >> > property >> > > > command for each partition. This is manual. >> > > > >> > > > 2. Set the logical operator property but wait till partition >> happens. >> > > This >> > > > will require partitioning to be triggered for which a StatsListener >> > needs >> > > > to be implemented. So if partitioning never happens this will never >> be >> > > > triggered. Also un-deploying and deploying a partition just to >> change a >> > > > property on the physical partition seems un-necessary >> > > > >> > > > 3. Implementing StatsListener to send operator request and not >> > > re-partition >> > > > seems to be appropriate way. >> > > > >> > > > Chandni >> > > > >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni < >> > > [email protected]> >> > > > wrote: >> > > > >> > > > > Priyanka you want to divide the value among existing partitions >> right >> > > and >> > > > > not necessarily change the number of partitions. To this end >> Ashwin >> > is >> > > > > suggesting makes sense. However different partitions can be at >> > > different >> > > > > windows and will see the value change in different windows. You >> need >> > a >> > > > > separate operator to receive this change and propagate it to the >> > > > partitions >> > > > > via data flow. Tim discussed this over the last weekend in a >> series >> > of >> > > > > emails. >> > > > > >> > > > > Thanks >> > > > > >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale < >> > > > > [email protected]> >> > > > > wrote: >> > > > > >> > > > > > Thanks for suggestions. >> > > > > > >> > > > > > Ashwin, The option you proposed is doable, but then each >> operator >> > > which >> > > > > > needs to address such use case will have to track the number of >> > > > > partitions. >> > > > > > So we were looking for some common mechanism. >> > > > > > Chandni, Let me try the res.operatorRequests, that could solve >> the >> > > > > problem. >> > > > > > >> > > > > > -Priyanka >> > > > > > >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh < >> > > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi Priyanka, >> > > > > > > >> > > > > > > As Isha mentioned you will can implement a StatsListener. >> However >> > > > > instead >> > > > > > > of setting res.repartitionRequired = true, you can add a >> > > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This >> will >> > > not >> > > > > > > re-deploy the instances. >> > > > > > > >> > > > > > > The OperatorRequest that you will need to implement will have >> to >> > > > change >> > > > > > the >> > > > > > > property value. >> > > > > > > >> > > > > > > You can refer to AutoMetricTest for an example. >> > > > > > > >> > > > > > > - Chandni >> > > > > > > >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta < >> > > > > > > [email protected]> wrote: >> > > > > > > >> > > > > > > > Why not maintain a variable containing the number of >> partitions >> > > in >> > > > > each >> > > > > > > > partition; and when you set the property, just divide it by >> > > number >> > > > of >> > > > > > > > partitions variable? Does it not solve the problem? >> > > > > > > > >> > > > > > > > Regards, >> > > > > > > > Ashwin. >> > > > > > > > >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale < >> > > > > > > [email protected] >> > > > > > > > > >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > I have an integer property named "intProp". This property >> is >> > > > > > applicable >> > > > > > > > to >> > > > > > > > > an operator A. I can set the property from dtcli using >> > > > > > > > > "set-operator-property operator-name property-name >> > > > property-value" >> > > > > > and >> > > > > > > > this >> > > > > > > > > will get applied to all instances of opeartor A. >> > > > > > > > > >> > > > > > > > > Now, I want to apply this property in a way that it's >> equally >> > > > > divided >> > > > > > > > > amongst all instances of operator A. For example, if >> > > intProp=100 >> > > > > and >> > > > > > > > > number of partitions=4 then intProp=25 should be set on >> each >> > > > > instance >> > > > > > > of >> > > > > > > > > operator A. >> > > > > > > > > >> > > > > > > > > Is there a way to achieve this? >> > > > > > > > > >> > > > > > > > > I have tried following approach to do this: I have >> overriden >> > > > > > > > > definePartitions of Operator, once we do partitioning I am >> > > > setting >> > > > > > > right >> > > > > > > > > value on each operator instance based on new partition >> count. >> > > The >> > > > > > > > > limitation here is that the value is applied during >> > > > > definePartitions. >> > > > > > > If >> > > > > > > > > the value is updated dynamically it won't be applied till >> > > > > > > > definePartitions >> > > > > > > > > is called again. >> > > > > > > > > >> > > > > > > > > -Priyanka >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > >> > > > > > > > Regards, >> > > > > > > > Ashwin. >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
