Currently, DAG topological changes are limited to repartitioning of existing operators after stats update and client directed logical plan changes via web services. We have seen the need for more dynamic plan changes especially from batch use cases and other scenarios.
I wanted to start a discussion around broadening this by increasing the scope of changes that can be done after say a stats update from only repartitioning and also allow more scenarios in which the changes can be done. After stats update for example, instead of only having the ability to trigger a repartition we could give the ability for the user code to change the plan. Plan changes can also be triggered after major events such as when a stram event gets created. In batch for example after shutdown is received from one stage another plan can be created and started. Thanks
