Here is the JIRA to track this https://malhar.atlassian.net/browse/APEX-163
On Mon, Sep 28, 2015 at 10:17 AM, Pramod Immaneni <[email protected]> wrote: > Apex support modification of operator properties at runtime but the > current implemenations has the following shortcomings. > > 1. Property is not set across all partitions on the same window as > individual partitions can be on different windows when property change is > initiated from client resulting in inconsistency of data for those windows. > I am being generous using the word inconsistent. > 2. Sometimes properties need to be set on more than one logical operators > at the same time to achieve the change the user is seeking. Today they will > be two separate changes happening on two different windows again resulting > in inconsistent data for some windows. These would need to happen as a > single transaction. > 3. If there is an operator failure before a committed checkpoint after an > operator property is dynamically changed the operator will restart with the > old property and the change will not be re-applied. > > Tim and myself did some brainstorming and we have a proposal to overcome > these shortcomings. The main problem in all the above cases is that the > property changes are happening out-of-band of data flow and hence > independent of windowing. The proposal is to bring the property change > request into the in-band dataflow so that they are handled consistently > with windowing and handled distributively. > > The idea is to inject a special property change tuple containing the > property changes and the identification information of the operator's they > affect into the dataflow at the input operator. The tuple will be injected > at window boundary after end window and before begin window and as this > tuple flows through the DAG the intended operators properties will be > modifed. They will all be modified consistently at the same window. The > tuple can contain more than one property changes for more than one logical > operators and the change will be applied consistently to the different > logical operators at the same window. In case of failure the replay of > tuples will ensure that the property change gets reapplied at the correct > window. > > Please give your feedback and input on what you think about this proposal. > > Thanks >
