Adding another operator to feed property changes to all the partitions requires more work. Same partition keys need to be assigned to all the partitions to receive all the property change triggers. Using a partitioner will result in re-deployment.
All the approaches seem to be an overkill for this simple task. Can we not simplify this for the users? Chandni On Wed, Sep 23, 2015 at 12:15 AM, Pramod Immaneni <[email protected]> wrote: > Typically the window difference wouldn't be large and if its happening its > probably indicator of a larger problem in the application anyway. > > On Wed, Sep 23, 2015 at 12:07 AM, Gaurav Gupta <[email protected]> > wrote: > > > Is that acceptable? as user may want to tune operator A now so that it > can > > catch up > > > > Thanks > > -Gaurav > > > > On Wed, Sep 23, 2015 at 12:04 AM, Pramod Immaneni < > [email protected]> > > wrote: > > > > > Yes. > > > > > > On Wed, Sep 23, 2015 at 12:03 AM, Gaurav Gupta <[email protected] > > > > > wrote: > > > > > > > Pramod, > > > > > > > > What happens if the Separate Operator that is used to propagate this > > > change > > > > is on say window 1000 and all the instances of Operator A are at > > 300..At > > > > this time user changes value of property. Then the new value will be > > > > delayed to Operator A instances by 700 windows > > > > > > > > Thanks > > > > -Gaurav > > > > > > > > On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta < > [email protected] > > > > > > > wrote: > > > > > > > > > Pramod , > > > > > > > > > > That is what I was thinking that Operator requests can also be > > received > > > > in > > > > > different windows...Looks like adding another operator is safe > > solution > > > > > > > > > > Thanks > > > > > -Gaurav > > > > > > > > > > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni < > > > > [email protected]> > > > > > wrote: > > > > > > > > > >> The operator request would also be received in different windows. > > > > >> > > > > >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale < > > > > >> [email protected]> > > > > >> wrote: > > > > >> > > > > >> > Promod, > > > > >> > > > > > >> > You are right about applying the value in same window. And yes I > > > don't > > > > >> want > > > > >> > to change number of partitions or even redeploy any partition > for > > > > this. > > > > >> > I need to explore on Chandini's suggestion of using > StatsListener. > > > > >> > > > > > >> > -Priyanka > > > > >> > > > > > >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni < > > > > >> [email protected]> > > > > >> > wrote: > > > > >> > > > > > >> > > I am not sure about priyanka's requirements but typically you > > > would > > > > >> want > > > > >> > to > > > > >> > > apply the value in the same window for all partitions for > > > > consistency > > > > >> > with > > > > >> > > the calculations. This could be done two ways using a separate > > > > logical > > > > >> > > operator feeding the value to all partitions so that the value > > > > >> appears in > > > > >> > > the same window or repartitioning the existing set and setting > > the > > > > >> value > > > > >> > > during repartitioning. In the latter case you are just > > > > repartitioning > > > > >> so > > > > >> > > that the partitions start from same point with the new value > and > > > not > > > > >> to > > > > >> > > change the number of partitions. > > > > >> > > > > > > >> > > Thanks > > > > >> > > > > > > >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh < > > > > >> [email protected] > > > > >> > > > > > > >> > > wrote: > > > > >> > > > > > > >> > > > Pramod, > > > > >> > > > > > > > >> > > > Tim's discussion was about setting a property on multiple > > > > operators > > > > >> > > > (different logical operators). > > > > >> > > > Priyanka's question is about different partitions of an > > operator > > > > >> > > > > > > > >> > > > Let's say they are 5 partitions of an operator and we need > to > > > > >> change an > > > > >> > > int > > > > >> > > > property. There are following ways: > > > > >> > > > > > > > >> > > > 1. Divide the int property by 5 and issue set-physical > > operator > > > > >> > property > > > > >> > > > command for each partition. This is manual. > > > > >> > > > > > > > >> > > > 2. Set the logical operator property but wait till partition > > > > >> happens. > > > > >> > > This > > > > >> > > > will require partitioning to be triggered for which a > > > > StatsListener > > > > >> > needs > > > > >> > > > to be implemented. So if partitioning never happens this > will > > > > never > > > > >> be > > > > >> > > > triggered. Also un-deploying and deploying a partition just > to > > > > >> change a > > > > >> > > > property on the physical partition seems un-necessary > > > > >> > > > > > > > >> > > > 3. Implementing StatsListener to send operator request and > not > > > > >> > > re-partition > > > > >> > > > seems to be appropriate way. > > > > >> > > > > > > > >> > > > Chandni > > > > >> > > > > > > > >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni < > > > > >> > > [email protected]> > > > > >> > > > wrote: > > > > >> > > > > > > > >> > > > > Priyanka you want to divide the value among existing > > > partitions > > > > >> right > > > > >> > > and > > > > >> > > > > not necessarily change the number of partitions. To this > end > > > > >> Ashwin > > > > >> > is > > > > >> > > > > suggesting makes sense. However different partitions can > be > > at > > > > >> > > different > > > > >> > > > > windows and will see the value change in different > windows. > > > You > > > > >> need > > > > >> > a > > > > >> > > > > separate operator to receive this change and propagate it > to > > > the > > > > >> > > > partitions > > > > >> > > > > via data flow. Tim discussed this over the last weekend > in a > > > > >> series > > > > >> > of > > > > >> > > > > emails. > > > > >> > > > > > > > > >> > > > > Thanks > > > > >> > > > > > > > > >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale < > > > > >> > > > > [email protected]> > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > > > Thanks for suggestions. > > > > >> > > > > > > > > > >> > > > > > Ashwin, The option you proposed is doable, but then each > > > > >> operator > > > > >> > > which > > > > >> > > > > > needs to address such use case will have to track the > > number > > > > of > > > > >> > > > > partitions. > > > > >> > > > > > So we were looking for some common mechanism. > > > > >> > > > > > Chandni, Let me try the res.operatorRequests, that could > > > solve > > > > >> the > > > > >> > > > > problem. > > > > >> > > > > > > > > > >> > > > > > -Priyanka > > > > >> > > > > > > > > > >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh < > > > > >> > > > [email protected]> > > > > >> > > > > > wrote: > > > > >> > > > > > > > > > >> > > > > > > Hi Priyanka, > > > > >> > > > > > > > > > > >> > > > > > > As Isha mentioned you will can implement a > > StatsListener. > > > > >> However > > > > >> > > > > instead > > > > >> > > > > > > of setting res.repartitionRequired = true, you can > add a > > > > >> > > > > > > StatsListener.OperatorRequest to > res.operatorsRequests. > > > This > > > > >> will > > > > >> > > not > > > > >> > > > > > > re-deploy the instances. > > > > >> > > > > > > > > > > >> > > > > > > The OperatorRequest that you will need to implement > will > > > > have > > > > >> to > > > > >> > > > change > > > > >> > > > > > the > > > > >> > > > > > > property value. > > > > >> > > > > > > > > > > >> > > > > > > You can refer to AutoMetricTest for an example. > > > > >> > > > > > > > > > > >> > > > > > > - Chandni > > > > >> > > > > > > > > > > >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta > < > > > > >> > > > > > > [email protected]> wrote: > > > > >> > > > > > > > > > > >> > > > > > > > Why not maintain a variable containing the number of > > > > >> partitions > > > > >> > > in > > > > >> > > > > each > > > > >> > > > > > > > partition; and when you set the property, just > divide > > it > > > > by > > > > >> > > number > > > > >> > > > of > > > > >> > > > > > > > partitions variable? Does it not solve the problem? > > > > >> > > > > > > > > > > > >> > > > > > > > Regards, > > > > >> > > > > > > > Ashwin. > > > > >> > > > > > > > > > > > >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale < > > > > >> > > > > > > [email protected] > > > > >> > > > > > > > > > > > > >> > > > > > > > wrote: > > > > >> > > > > > > > > > > > >> > > > > > > > > I have an integer property named "intProp". This > > > > property > > > > >> is > > > > >> > > > > > applicable > > > > >> > > > > > > > to > > > > >> > > > > > > > > an operator A. I can set the property from dtcli > > using > > > > >> > > > > > > > > "set-operator-property operator-name property-name > > > > >> > > > property-value" > > > > >> > > > > > and > > > > >> > > > > > > > this > > > > >> > > > > > > > > will get applied to all instances of opeartor A. > > > > >> > > > > > > > > > > > > >> > > > > > > > > Now, I want to apply this property in a way that > > it's > > > > >> equally > > > > >> > > > > divided > > > > >> > > > > > > > > amongst all instances of operator A. For example, > > if > > > > >> > > intProp=100 > > > > >> > > > > and > > > > >> > > > > > > > > number of partitions=4 then intProp=25 should be > set > > > on > > > > >> each > > > > >> > > > > instance > > > > >> > > > > > > of > > > > >> > > > > > > > > operator A. > > > > >> > > > > > > > > > > > > >> > > > > > > > > Is there a way to achieve this? > > > > >> > > > > > > > > > > > > >> > > > > > > > > I have tried following approach to do this: I have > > > > >> overriden > > > > >> > > > > > > > > definePartitions of Operator, once we do > > partitioning > > > I > > > > am > > > > >> > > > setting > > > > >> > > > > > > right > > > > >> > > > > > > > > value on each operator instance based on new > > partition > > > > >> count. > > > > >> > > The > > > > >> > > > > > > > > limitation here is that the value is applied > during > > > > >> > > > > definePartitions. > > > > >> > > > > > > If > > > > >> > > > > > > > > the value is updated dynamically it won't be > applied > > > > till > > > > >> > > > > > > > definePartitions > > > > >> > > > > > > > > is called again. > > > > >> > > > > > > > > > > > > >> > > > > > > > > -Priyanka > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > -- > > > > >> > > > > > > > > > > > >> > > > > > > > Regards, > > > > >> > > > > > > > Ashwin. > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
