Yes we should explore supporting this in the engine. On Wed, Sep 23, 2015 at 9:22 AM, Chandni Singh <[email protected]> wrote:
> Adding another operator to feed property changes to all the partitions > requires more work. Same partition keys need to be assigned to all the > partitions to receive all the property change triggers. > Using a partitioner will result in re-deployment. > > All the approaches seem to be an overkill for this simple task. Can we not > simplify this for the users? > > Chandni > > On Wed, Sep 23, 2015 at 12:15 AM, Pramod Immaneni <[email protected]> > wrote: > > > Typically the window difference wouldn't be large and if its happening > its > > probably indicator of a larger problem in the application anyway. > > > > On Wed, Sep 23, 2015 at 12:07 AM, Gaurav Gupta <[email protected]> > > wrote: > > > > > Is that acceptable? as user may want to tune operator A now so that it > > can > > > catch up > > > > > > Thanks > > > -Gaurav > > > > > > On Wed, Sep 23, 2015 at 12:04 AM, Pramod Immaneni < > > [email protected]> > > > wrote: > > > > > > > Yes. > > > > > > > > On Wed, Sep 23, 2015 at 12:03 AM, Gaurav Gupta < > [email protected] > > > > > > > wrote: > > > > > > > > > Pramod, > > > > > > > > > > What happens if the Separate Operator that is used to propagate > this > > > > change > > > > > is on say window 1000 and all the instances of Operator A are at > > > 300..At > > > > > this time user changes value of property. Then the new value will > be > > > > > delayed to Operator A instances by 700 windows > > > > > > > > > > Thanks > > > > > -Gaurav > > > > > > > > > > On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Pramod , > > > > > > > > > > > > That is what I was thinking that Operator requests can also be > > > received > > > > > in > > > > > > different windows...Looks like adding another operator is safe > > > solution > > > > > > > > > > > > Thanks > > > > > > -Gaurav > > > > > > > > > > > > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni < > > > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > >> The operator request would also be received in different > windows. > > > > > >> > > > > > >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale < > > > > > >> [email protected]> > > > > > >> wrote: > > > > > >> > > > > > >> > Promod, > > > > > >> > > > > > > >> > You are right about applying the value in same window. And > yes I > > > > don't > > > > > >> want > > > > > >> > to change number of partitions or even redeploy any partition > > for > > > > > this. > > > > > >> > I need to explore on Chandini's suggestion of using > > StatsListener. > > > > > >> > > > > > > >> > -Priyanka > > > > > >> > > > > > > >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni < > > > > > >> [email protected]> > > > > > >> > wrote: > > > > > >> > > > > > > >> > > I am not sure about priyanka's requirements but typically > you > > > > would > > > > > >> want > > > > > >> > to > > > > > >> > > apply the value in the same window for all partitions for > > > > > consistency > > > > > >> > with > > > > > >> > > the calculations. This could be done two ways using a > separate > > > > > logical > > > > > >> > > operator feeding the value to all partitions so that the > value > > > > > >> appears in > > > > > >> > > the same window or repartitioning the existing set and > setting > > > the > > > > > >> value > > > > > >> > > during repartitioning. In the latter case you are just > > > > > repartitioning > > > > > >> so > > > > > >> > > that the partitions start from same point with the new value > > and > > > > not > > > > > >> to > > > > > >> > > change the number of partitions. > > > > > >> > > > > > > > >> > > Thanks > > > > > >> > > > > > > > >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh < > > > > > >> [email protected] > > > > > >> > > > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > Pramod, > > > > > >> > > > > > > > > >> > > > Tim's discussion was about setting a property on multiple > > > > > operators > > > > > >> > > > (different logical operators). > > > > > >> > > > Priyanka's question is about different partitions of an > > > operator > > > > > >> > > > > > > > > >> > > > Let's say they are 5 partitions of an operator and we need > > to > > > > > >> change an > > > > > >> > > int > > > > > >> > > > property. There are following ways: > > > > > >> > > > > > > > > >> > > > 1. Divide the int property by 5 and issue set-physical > > > operator > > > > > >> > property > > > > > >> > > > command for each partition. This is manual. > > > > > >> > > > > > > > > >> > > > 2. Set the logical operator property but wait till > partition > > > > > >> happens. > > > > > >> > > This > > > > > >> > > > will require partitioning to be triggered for which a > > > > > StatsListener > > > > > >> > needs > > > > > >> > > > to be implemented. So if partitioning never happens this > > will > > > > > never > > > > > >> be > > > > > >> > > > triggered. Also un-deploying and deploying a partition > just > > to > > > > > >> change a > > > > > >> > > > property on the physical partition seems un-necessary > > > > > >> > > > > > > > > >> > > > 3. Implementing StatsListener to send operator request and > > not > > > > > >> > > re-partition > > > > > >> > > > seems to be appropriate way. > > > > > >> > > > > > > > > >> > > > Chandni > > > > > >> > > > > > > > > >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni < > > > > > >> > > [email protected]> > > > > > >> > > > wrote: > > > > > >> > > > > > > > > >> > > > > Priyanka you want to divide the value among existing > > > > partitions > > > > > >> right > > > > > >> > > and > > > > > >> > > > > not necessarily change the number of partitions. To this > > end > > > > > >> Ashwin > > > > > >> > is > > > > > >> > > > > suggesting makes sense. However different partitions can > > be > > > at > > > > > >> > > different > > > > > >> > > > > windows and will see the value change in different > > windows. > > > > You > > > > > >> need > > > > > >> > a > > > > > >> > > > > separate operator to receive this change and propagate > it > > to > > > > the > > > > > >> > > > partitions > > > > > >> > > > > via data flow. Tim discussed this over the last weekend > > in a > > > > > >> series > > > > > >> > of > > > > > >> > > > > emails. > > > > > >> > > > > > > > > > >> > > > > Thanks > > > > > >> > > > > > > > > > >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale < > > > > > >> > > > > [email protected]> > > > > > >> > > > > wrote: > > > > > >> > > > > > > > > > >> > > > > > Thanks for suggestions. > > > > > >> > > > > > > > > > > >> > > > > > Ashwin, The option you proposed is doable, but then > each > > > > > >> operator > > > > > >> > > which > > > > > >> > > > > > needs to address such use case will have to track the > > > number > > > > > of > > > > > >> > > > > partitions. > > > > > >> > > > > > So we were looking for some common mechanism. > > > > > >> > > > > > Chandni, Let me try the res.operatorRequests, that > could > > > > solve > > > > > >> the > > > > > >> > > > > problem. > > > > > >> > > > > > > > > > > >> > > > > > -Priyanka > > > > > >> > > > > > > > > > > >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh < > > > > > >> > > > [email protected]> > > > > > >> > > > > > wrote: > > > > > >> > > > > > > > > > > >> > > > > > > Hi Priyanka, > > > > > >> > > > > > > > > > > > >> > > > > > > As Isha mentioned you will can implement a > > > StatsListener. > > > > > >> However > > > > > >> > > > > instead > > > > > >> > > > > > > of setting res.repartitionRequired = true, you can > > add a > > > > > >> > > > > > > StatsListener.OperatorRequest to > > res.operatorsRequests. > > > > This > > > > > >> will > > > > > >> > > not > > > > > >> > > > > > > re-deploy the instances. > > > > > >> > > > > > > > > > > > >> > > > > > > The OperatorRequest that you will need to implement > > will > > > > > have > > > > > >> to > > > > > >> > > > change > > > > > >> > > > > > the > > > > > >> > > > > > > property value. > > > > > >> > > > > > > > > > > > >> > > > > > > You can refer to AutoMetricTest for an example. > > > > > >> > > > > > > > > > > > >> > > > > > > - Chandni > > > > > >> > > > > > > > > > > > >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra > Putta > > < > > > > > >> > > > > > > [email protected]> wrote: > > > > > >> > > > > > > > > > > > >> > > > > > > > Why not maintain a variable containing the number > of > > > > > >> partitions > > > > > >> > > in > > > > > >> > > > > each > > > > > >> > > > > > > > partition; and when you set the property, just > > divide > > > it > > > > > by > > > > > >> > > number > > > > > >> > > > of > > > > > >> > > > > > > > partitions variable? Does it not solve the > problem? > > > > > >> > > > > > > > > > > > > >> > > > > > > > Regards, > > > > > >> > > > > > > > Ashwin. > > > > > >> > > > > > > > > > > > > >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale < > > > > > >> > > > > > > [email protected] > > > > > >> > > > > > > > > > > > > > >> > > > > > > > wrote: > > > > > >> > > > > > > > > > > > > >> > > > > > > > > I have an integer property named "intProp". This > > > > > property > > > > > >> is > > > > > >> > > > > > applicable > > > > > >> > > > > > > > to > > > > > >> > > > > > > > > an operator A. I can set the property from dtcli > > > using > > > > > >> > > > > > > > > "set-operator-property operator-name > property-name > > > > > >> > > > property-value" > > > > > >> > > > > > and > > > > > >> > > > > > > > this > > > > > >> > > > > > > > > will get applied to all instances of opeartor A. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Now, I want to apply this property in a way that > > > it's > > > > > >> equally > > > > > >> > > > > divided > > > > > >> > > > > > > > > amongst all instances of operator A. For > example, > > > if > > > > > >> > > intProp=100 > > > > > >> > > > > and > > > > > >> > > > > > > > > number of partitions=4 then intProp=25 should be > > set > > > > on > > > > > >> each > > > > > >> > > > > instance > > > > > >> > > > > > > of > > > > > >> > > > > > > > > operator A. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Is there a way to achieve this? > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > I have tried following approach to do this: I > have > > > > > >> overriden > > > > > >> > > > > > > > > definePartitions of Operator, once we do > > > partitioning > > > > I > > > > > am > > > > > >> > > > setting > > > > > >> > > > > > > right > > > > > >> > > > > > > > > value on each operator instance based on new > > > partition > > > > > >> count. > > > > > >> > > The > > > > > >> > > > > > > > > limitation here is that the value is applied > > during > > > > > >> > > > > definePartitions. > > > > > >> > > > > > > If > > > > > >> > > > > > > > > the value is updated dynamically it won't be > > applied > > > > > till > > > > > >> > > > > > > > definePartitions > > > > > >> > > > > > > > > is called again. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > -Priyanka > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > -- > > > > > >> > > > > > > > > > > > > >> > > > > > > > Regards, > > > > > >> > > > > > > > Ashwin. > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
