GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/2414
[FLINK-4341] Let idle consumer subtasks emit max value watermarks and fail
on resharding
This is a short-term fix, until the min-watermark service for the
JobManager described in the JIRA discussion is available.
The way this fix works is that we let idle subtasks that initially don't
get assigned shards emit a `Long.MAX_VALUE` watermark. Also, we _only fail hard
if an idle subtask_ is assigned new shards when resharding happens, to avoid
messing up the watermarks. So, if all subtasks are not initially idle on
startup (i.e., when total number of shards > consumer parallelism), the Kinesis
consumer can still transparently handle resharding like before without failing.
I've tested exactly-once with our manual tests (with and w/o resharding)
and the fix works nicely, still retaining exactly-once guarantee despite
non-transparency. However, I'm a bit unsure on how to test if the unbounded
state with window operators is also fixed with this change, so we're still yet
to clarify this.
R: @rmetzger and @aljoscha for review. Thanks in advance!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-4341
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2414.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2414
----
commit bc8e50d99be745300f7418c58e9d30abc5469ba3
Author: Gordon Tai <[email protected]>
Date: 2016-08-24T08:38:06Z
[FLINK-4341] Let idle consumer subtasks emit max value watermarks and fail
on resharding
This no longer allows the Kinesis consumer to transparently handle
resharding.
This is a short-term workaround until we have a min-watermark notification
service available in the JobManager.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
