Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
These are some good points @StephanEwen, thanks for checking it.
How about the following, regarding each issue:
- `Concurrent Checkpoints`: Looks like an improvement but I can sure do it
in this PR if it is a crucial one. Can you elaborate a bit more or point me
out to other concurrent checkpointing operator state examples to get an idea of
how you want to do it?
- `Reconfiguration` : Sounds interesting...but I am not really aware of it
from the devlist. If it is simple enough I could add support for it here.
Otherwise I would suggest we address this in a seperate JIRA and PR as an
improvement. Is there a design document on how we plan to achieve
reconfiguration and repartitioning for operator state specifically somewhere?
- `At-most-once blocking queue` : It is obvious from my previous comments
that I do not approve this part, but that is something we already got rid of in
[FLIP-15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-15+Scoped+Loops+and+Job+Termination)
already
([this](https://github.com/FouadMA/flink/commit/9adaac435bcaf3552afe564c739d4e8fd79c433b)
commit). How about we address this together with the deadlocks in FLIP-15?
- `Deadlocks`: I like the elastic spilling channel idea to resolve
deadlocks. I need time to dig a bit more into this and make sure we solve
deadlocks and not just improve. Is it ok with you if we address that in
[FLIP-15](https://cwiki.apache.org/confluence/display/FLINK/FLIP-15+Scoped+Loops+and+Job+Termination)?
I need more time for this part, plus, we need to combine the absense of
expiring queues with a proper termination algorithm (otherwise we just solve
the deadlocks and the jobs never terminate).
What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
