[jira] [Created] (FLINK-21505) Enforce common savepoint format at the operator level
Aljoscha Krettek created FLINK-21505: Summary: Enforce common savepoint format at the operator level Key: FLINK-21505 URL: https://issues.apache.org/jira/browse/FLINK-21505 Project: Flink Issue Type: Sub-task Components: Runtime / Checkpointing Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Currently, we are relying on the fact that all keyed backends would use the same strategy for savepoints. We should be forcing them at the API level to ensure that all exiting and future state backends will creat savepoints in the same format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20976) Unify Binary format for Keyed State savepoints
[ https://issues.apache.org/jira/browse/FLINK-20976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-20976: Assignee: Dawid Wysakowicz > Unify Binary format for Keyed State savepoints > -- > > Key: FLINK-20976 > URL: https://issues.apache.org/jira/browse/FLINK-20976 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Fix For: 1.13.0 > > > The main goal of this proposal is the following: > * Unify across all state backends a savepoint format for keyed state that is > more future-proof and applicable for potential new state backends. Checkpoint > formats, by definition, are still allowed to be backend specific. > * Make it possible to switch a state backend via a savepoint > * Rework abstractions related to snapshots and restoring, to reduce the > overhead and code duplication when attempting to implement a new state > backend. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20918) Avoid excessive flush of Hadoop output stream
[ https://issues.apache.org/jira/browse/FLINK-20918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278767#comment-17278767 ] Aljoscha Krettek commented on FLINK-20918: -- Is there an observed real-world impact of this? I would be cautious with just removing the call since some File Systems might have unexpected implementations. After all, the interface does have the two methods. > Avoid excessive flush of Hadoop output stream > - > > Key: FLINK-20918 > URL: https://issues.apache.org/jira/browse/FLINK-20918 > Project: Flink > Issue Type: Bug > Components: Connectors / Hadoop Compatibility, FileSystems >Affects Versions: 1.11.3, 1.12.0 >Reporter: Paul Lin >Priority: Major > Labels: pull-request-available > > [HadoopRecoverableFsDataOutputStream#sync|https://github.com/apache/flink/blob/67d167ccd45046fc5ed222ac1f1e3ba5e6ec434b/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableFsDataOutputStream.java#L123] > calls both `hflush` and `hsync`, whereas `hsync` is an enhanced version of > `hflush`. We should remove the `hflush` call to avoid the excessive flush. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21151) Extract common full-snapshot writer from RocksDB full-snapshot strategy
[ https://issues.apache.org/jira/browse/FLINK-21151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-21151. Fix Version/s: 1.13.0 Resolution: Implemented master: 9a74bd63413e833f59503c895957840b84995216 and commits around it > Extract common full-snapshot writer from RocksDB full-snapshot strategy > --- > > Key: FLINK-21151 > URL: https://issues.apache.org/jira/browse/FLINK-21151 > Project: Flink > Issue Type: Sub-task > Components: Runtime / State Backends >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > As described in FLIP-41, the RocksDB full-snapshot format will serve as the > common, unified savepoint format. We need to extract the common parts and > make them reusable by other state backends. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17274459#comment-17274459
]
Aljoscha Krettek commented on FLINK-21132:
--
I'll try and expand on what I wrote earlier. The TL;DR is that I still think
sending an {{endOfInput()}} to operators on a stop-with-savepoint is a bug.
I differentiate between two different end-of-inputs: 1) _physical_
end-of-input, and 2) _logical/semantical_ end-of-input. The former is signalled
when, for example, a network connection is being shut down. The latter happens
(should happen) when (bounded) sources have no more data to read and this
information propagates through the pipeline. The motivation for introducing
{{endOfInput()}} were things like hash-join in the SQL runner where an operator
would read from the build side until getting an end-of-input, at which point it
would switch over to reading from the probe side. With these use cases in mind
sending an {{endOfInput()}} is a bug. The same is true for sinks, which will do
some bookkeeping based on knowing that all the input data has been read.
I don't see use cases where operators would be interested in being notified of
the physical end-of-input right now. I could be wrong, of course.
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Assignee: Roman Khachatryan
>Priority: Major
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-12461) Document binary compatibility situation with Scala beyond 2.12.8
[ https://issues.apache.org/jira/browse/FLINK-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-12461. Resolution: Workaround The workaround is compiling manually for the specific Scala version. > Document binary compatibility situation with Scala beyond 2.12.8 > > > Key: FLINK-12461 > URL: https://issues.apache.org/jira/browse/FLINK-12461 > Project: Flink > Issue Type: Bug > Components: API / Scala >Affects Versions: 1.8.0 >Reporter: Gaël Renoux >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0, 1.12.2 > > > When using 1.8 with Scala 2.12.8 and trying to parse a scala.Map (not a > java.util.Map), I get a: java.lang.ClassNotFoundException: > scala.math.Ordering$$anon$9 > To reproduce: > - start with the Scala Maven archetype > (org.apache.flink:flink-quickstart-scala:1.8.0) > - in the POM, set the scala.version to 2.12.8 and the scala.binary.version > to 2.12 > - in StreamingJob, add: env.fromElements[Map[String, Int]]() > It works with Scala 2.12.7 (well, without putting anything in the job, it > fails with "No operators defined in streaming topology", which is expected). > I suspect this is linked to the binary incompatiblity of 2.12.8 with 2.12.7 > (see the release note of 2.12.8), so compiling Flink with 2.12.8 instead of > 2.12.7 might be enough (although it might stop working with 2.12.7?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-12461) Document binary compatibility situation with Scala beyond 2.12.8
[ https://issues.apache.org/jira/browse/FLINK-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-12461: Assignee: Aljoscha Krettek > Document binary compatibility situation with Scala beyond 2.12.8 > > > Key: FLINK-12461 > URL: https://issues.apache.org/jira/browse/FLINK-12461 > Project: Flink > Issue Type: Bug > Components: API / Scala >Affects Versions: 1.8.0 >Reporter: Gaël Renoux >Assignee: Aljoscha Krettek >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0, 1.12.2 > > > When using 1.8 with Scala 2.12.8 and trying to parse a scala.Map (not a > java.util.Map), I get a: java.lang.ClassNotFoundException: > scala.math.Ordering$$anon$9 > To reproduce: > - start with the Scala Maven archetype > (org.apache.flink:flink-quickstart-scala:1.8.0) > - in the POM, set the scala.version to 2.12.8 and the scala.binary.version > to 2.12 > - in StreamingJob, add: env.fromElements[Map[String, Int]]() > It works with Scala 2.12.7 (well, without putting anything in the job, it > fails with "No operators defined in streaming topology", which is expected). > I suspect this is linked to the binary incompatiblity of 2.12.8 with 2.12.7 > (see the release note of 2.12.8), so compiling Flink with 2.12.8 instead of > 2.12.7 might be enough (although it might stop working with 2.12.7?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12461) Document binary compatibility situation with Scala beyond 2.12.8
[ https://issues.apache.org/jira/browse/FLINK-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-12461: - Summary: Document binary compatibility situation with Scala beyond 2.12.8 (was: Flink 1.8 not working with Scala 2.12.8) > Document binary compatibility situation with Scala beyond 2.12.8 > > > Key: FLINK-12461 > URL: https://issues.apache.org/jira/browse/FLINK-12461 > Project: Flink > Issue Type: Bug > Components: API / Scala >Affects Versions: 1.8.0 >Reporter: Gaël Renoux >Priority: Major > Labels: pull-request-available > > When using 1.8 with Scala 2.12.8 and trying to parse a scala.Map (not a > java.util.Map), I get a: java.lang.ClassNotFoundException: > scala.math.Ordering$$anon$9 > To reproduce: > - start with the Scala Maven archetype > (org.apache.flink:flink-quickstart-scala:1.8.0) > - in the POM, set the scala.version to 2.12.8 and the scala.binary.version > to 2.12 > - in StreamingJob, add: env.fromElements[Map[String, Int]]() > It works with Scala 2.12.7 (well, without putting anything in the job, it > fails with "No operators defined in streaming topology", which is expected). > I suspect this is linked to the binary incompatiblity of 2.12.8 with 2.12.7 > (see the release note of 2.12.8), so compiling Flink with 2.12.8 instead of > 2.12.7 might be enough (although it might stop working with 2.12.7?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12461) Document binary compatibility situation with Scala beyond 2.12.8
[ https://issues.apache.org/jira/browse/FLINK-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-12461: - Fix Version/s: 1.12.2 1.13.0 > Document binary compatibility situation with Scala beyond 2.12.8 > > > Key: FLINK-12461 > URL: https://issues.apache.org/jira/browse/FLINK-12461 > Project: Flink > Issue Type: Bug > Components: API / Scala >Affects Versions: 1.8.0 >Reporter: Gaël Renoux >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0, 1.12.2 > > > When using 1.8 with Scala 2.12.8 and trying to parse a scala.Map (not a > java.util.Map), I get a: java.lang.ClassNotFoundException: > scala.math.Ordering$$anon$9 > To reproduce: > - start with the Scala Maven archetype > (org.apache.flink:flink-quickstart-scala:1.8.0) > - in the POM, set the scala.version to 2.12.8 and the scala.binary.version > to 2.12 > - in StreamingJob, add: env.fromElements[Map[String, Int]]() > It works with Scala 2.12.7 (well, without putting anything in the job, it > fails with "No operators defined in streaming topology", which is expected). > I suspect this is linked to the binary incompatiblity of 2.12.8 with 2.12.7 > (see the release note of 2.12.8), so compiling Flink with 2.12.8 instead of > 2.12.7 might be enough (although it might stop working with 2.12.7?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12461) Flink 1.8 not working with Scala 2.12.8
[ https://issues.apache.org/jira/browse/FLINK-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273629#comment-17273629 ] Aljoscha Krettek commented on FLINK-12461: -- I added an explanation of the situation to the docs. I'd close the issue now, because I don't think there's much we can do here still. What do you think? Added explanation in commits: - master: bbee6cd749f0ca462db47fb9feaac27538748112 - release-1.12: 0e49396853ef41d83dd8591bce9a44b99428535d > Flink 1.8 not working with Scala 2.12.8 > --- > > Key: FLINK-12461 > URL: https://issues.apache.org/jira/browse/FLINK-12461 > Project: Flink > Issue Type: Bug > Components: API / Scala >Affects Versions: 1.8.0 >Reporter: Gaël Renoux >Priority: Major > Labels: pull-request-available > > When using 1.8 with Scala 2.12.8 and trying to parse a scala.Map (not a > java.util.Map), I get a: java.lang.ClassNotFoundException: > scala.math.Ordering$$anon$9 > To reproduce: > - start with the Scala Maven archetype > (org.apache.flink:flink-quickstart-scala:1.8.0) > - in the POM, set the scala.version to 2.12.8 and the scala.binary.version > to 2.12 > - in StreamingJob, add: env.fromElements[Map[String, Int]]() > It works with Scala 2.12.7 (well, without putting anything in the job, it > fails with "No operators defined in streaming topology", which is expected). > I suspect this is linked to the binary incompatiblity of 2.12.8 with 2.12.7 > (see the release note of 2.12.8), so compiling Flink with 2.12.8 instead of > 2.12.7 might be enough (although it might stop working with 2.12.7?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272921#comment-17272921
]
Aljoscha Krettek commented on FLINK-21132:
--
It seems to me we just need to make sure that {{endOfInput()}} is not called
for the stop-with-savepoint case. That method must only be called when it is
*truly* the end of input, that is when the source and all the predecessor
operators have shut down. What do you think? With this, the Jira Issue should
be easy enough to fix.
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Priority: Major
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21106) google-java-format Intellij Plugin 1.7.0.5 causes UnsupportedOperationException in IntelliJ
[
https://issues.apache.org/jira/browse/FLINK-21106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272732#comment-17272732
]
Aljoscha Krettek commented on FLINK-21106:
--
Agreed!
> google-java-format Intellij Plugin 1.7.0.5 causes
> UnsupportedOperationException in IntelliJ
> ---
>
> Key: FLINK-21106
> URL: https://issues.apache.org/jira/browse/FLINK-21106
> Project: Flink
> Issue Type: Bug
>Reporter: Matthias
>Priority: Minor
> Attachments: google-java-format-1.7-patched.zip
>
>
> There's a problem with {{google-java-format}} Intellij plugin version
> {{1.7.0.5}} that causes an {{UnsupportedOperationException}} when creating a
> new Java class file. Besides the exception, an error dialog pops up and the
> newly created file is not properly formatted. A simple reformat solves the
> issue.
> This problem is caused by a bug that got fixed in the {{google-java-format}}
> plugin's codebase in
> [45fb41a|https://github.com/google/google-java-format/commit/45fb41a7bac3dfe0726601ceb87d1c17bbf494ec].
> Unfortunately, this fix got released with the plugin version {{1.8.0.1}}
> which we cannot upgrade to due to our limitations on sticking to Java 8 for
> now (see FLINK-20803).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21153) yarn-per-job deployment target ignores yarn options
[
https://issues.apache.org/jira/browse/FLINK-21153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272731#comment-17272731
]
Aljoscha Krettek commented on FLINK-21153:
--
I agree with [~fly_in_gis], the behaviour of the {{GenericCLI}} is different
(simpler) and eventually the special behavior we have for YARN should be
removed.
> yarn-per-job deployment target ignores yarn options
> ---
>
> Key: FLINK-21153
> URL: https://issues.apache.org/jira/browse/FLINK-21153
> Project: Flink
> Issue Type: Bug
> Components: Command Line Client, Deployment / YARN
>Affects Versions: 1.13.0, 1.12.1
>Reporter: Till Rohrmann
>Priority: Major
> Labels: usability
> Fix For: 1.13.0, 1.12.2
>
>
> While looking into the problem reported in FLINK-6949, I stumbled across an
> odd behaviour of Flink. I tried to deploy a Flink cluster on Yarn and ship
> some files to the cluster. Only the first command successfully shipped the
> additional files to the cluster:
> 1) {{bin/flink run -p 1 --yarnship ../flink-test-job/cluster -m yarn-cluster
> ../flink-test-job/target/flink-test-job-1.0-SNAPSHOT.jar}}
> 2) {{bin/flink run -p 1 --yarnship ../flink-test-job/cluster -t yarn-per-job
> ../flink-test-job/target/flink-test-job-1.0-SNAPSHOT.jar}}
> The problem seems to be that the second command does not activate the
> {{FlinkYarnSessionCli}} but uses the {{GenericCLI}}.
> [~kkl0u], [~aljoscha], [~tison] what is the intended behaviour in this case.
> I always thought that {{-m yarn-cluster}} and {{-t yarn-per-job}} would be
> equivalent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21151) Extract common full-snapshot writer from RocksDB full-snapshot strategy
Aljoscha Krettek created FLINK-21151: Summary: Extract common full-snapshot writer from RocksDB full-snapshot strategy Key: FLINK-21151 URL: https://issues.apache.org/jira/browse/FLINK-21151 Project: Flink Issue Type: Sub-task Components: Runtime / State Backends Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek As described in FLIP-41, the RocksDB full-snapshot format will serve as the common, unified savepoint format. We need to extract the common parts and make them reusable by other state backends. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21106) google-java-format Intellij Plugin 1.7.0.5 causes UnsupportedOperationException in IntelliJ
[
https://issues.apache.org/jira/browse/FLINK-21106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271520#comment-17271520
]
Aljoscha Krettek commented on FLINK-21106:
--
Side note: I'm just using the most recent IntelliJ plugin version and it's fine
for all of my code so far. For the rare case where more recent versions of the
plugin produce formatting that is incompatible with 1.7.5 our CI would catch it
and I can run {{mvn spotless:apply}} to fix it.
> google-java-format Intellij Plugin 1.7.0.5 causes
> UnsupportedOperationException in IntelliJ
> ---
>
> Key: FLINK-21106
> URL: https://issues.apache.org/jira/browse/FLINK-21106
> Project: Flink
> Issue Type: Bug
>Reporter: Matthias
>Priority: Minor
> Attachments: google-java-format-1.7-patched.zip
>
>
> There's a problem with {{google-java-format}} Intellij plugin version
> {{1.7.0.5}} that causes an {{UnsupportedOperationException}} when creating a
> new Java class file. Besides the exception, an error dialog pops up and the
> newly created file is not properly formatted. A simple reformat solves the
> issue.
> This problem is caused by a bug that got fixed in the {{google-java-format}}
> plugin's codebase in
> [45fb41a|https://github.com/google/google-java-format/commit/45fb41a7bac3dfe0726601ceb87d1c17bbf494ec].
> Unfortunately, this fix got released with the plugin version {{1.8.0.1}}
> which we cannot upgrade to due to our limitations on sticking to Java 8 for
> now (see FLINK-20803).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21133) FLIP-27 Source does not work with synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-21133:
-
Component/s: Runtime / Checkpointing
> FLIP-27 Source does not work with synchronous savepoint
> ---
>
> Key: FLINK-21133
> URL: https://issues.apache.org/jira/browse/FLINK-21133
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Runtime / Checkpointing
>Affects Versions: 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Priority: Major
>
> I have pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. {{SavepointITCase.testStopSavepointWithFlip27Source}}
> failed due to timeout.
> See also FLINK-21132 and
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21133) FLIP-27 Source does not work with synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-21133:
-
Component/s: API / Core
> FLIP-27 Source does not work with synchronous savepoint
> ---
>
> Key: FLINK-21133
> URL: https://issues.apache.org/jira/browse/FLINK-21133
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Priority: Major
>
> I have pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. {{SavepointITCase.testStopSavepointWithFlip27Source}}
> failed due to timeout.
> See also FLINK-21132 and
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]..
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-12461) Flink 1.8 not working with Scala 2.12.8
[ https://issues.apache.org/jira/browse/FLINK-12461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17270046#comment-17270046 ] Aljoscha Krettek commented on FLINK-12461: -- This should be related to my first comment. The problem is that there is no binary compatibility between Scala 2.12.x versions after 2.12.8 and our binary compatibility check will discover that. To fix this, you can disable japicmp locally for your build and see if everything else works out fine. However, I don't think we can do releases with this, because we would have to do releases for individual Scala 2.12.x versions, which seems a bit overkill. The situation with Scala is not ideal here. > Flink 1.8 not working with Scala 2.12.8 > --- > > Key: FLINK-12461 > URL: https://issues.apache.org/jira/browse/FLINK-12461 > Project: Flink > Issue Type: Bug > Components: API / Scala >Affects Versions: 1.8.0 >Reporter: Gaël Renoux >Priority: Major > > When using 1.8 with Scala 2.12.8 and trying to parse a scala.Map (not a > java.util.Map), I get a: java.lang.ClassNotFoundException: > scala.math.Ordering$$anon$9 > To reproduce: > - start with the Scala Maven archetype > (org.apache.flink:flink-quickstart-scala:1.8.0) > - in the POM, set the scala.version to 2.12.8 and the scala.binary.version > to 2.12 > - in StreamingJob, add: env.fromElements[Map[String, Int]]() > It works with Scala 2.12.7 (well, without putting anything in the job, it > fails with "No operators defined in streaming topology", which is expected). > I suspect this is linked to the binary incompatiblity of 2.12.8 with 2.12.7 > (see the release note of 2.12.8), so compiling Flink with 2.12.8 instead of > 2.12.7 might be enough (although it might stop working with 2.12.7?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21060) Could not create the DispatcherResourceManagerComponent
[ https://issues.apache.org/jira/browse/FLINK-21060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-21060: - Component/s: (was: API / Core) Table SQL / Ecosystem > Could not create the DispatcherResourceManagerComponent > --- > > Key: FLINK-21060 > URL: https://issues.apache.org/jira/browse/FLINK-21060 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Affects Versions: 1.12.0 > Environment: flink: 1.12.0 > hive: 3.1.2 >Reporter: Spongebob >Priority: Major > Attachments: yarn.log > > > I set multi sink to hive table in flink application then deploy it on yarn, > if I run with detached mode, the application can run successful but sink > nothing. And if I run without detached mode, the application would throw this > exception: Could not create the DispatcherResourceManagerComponent. > Attachment is the log of this application run without detached mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20517) Support mixed keyed/non-keyed operations in BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267327#comment-17267327 ] Aljoscha Krettek commented on FLINK-20517: -- release-1.12: 5c6d190cda81fcc3acc9eb4dfc1e30cb8d381d99 > Support mixed keyed/non-keyed operations in BATCH execution mode > > > Key: FLINK-20517 > URL: https://issues.apache.org/jira/browse/FLINK-20517 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Dawid Wysakowicz >Assignee: Aljoscha Krettek >Priority: Critical > Labels: pull-request-available > Fix For: 1.13.0, 1.12.2 > > > Using the batch state backend requires that the inputs are sorted by a key. > Right now when sorting, we sort either all of the inputs or none. It is > because we expect that the same key is consumed from all inputs at the same > time. > In the process of implementing FLINK-20491 we can also add support for mixed > non-keyed/keyed operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20517) Support mixed keyed/non-keyed operations in BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20517: - Fix Version/s: 1.12.2 > Support mixed keyed/non-keyed operations in BATCH execution mode > > > Key: FLINK-20517 > URL: https://issues.apache.org/jira/browse/FLINK-20517 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Dawid Wysakowicz >Assignee: Aljoscha Krettek >Priority: Critical > Labels: pull-request-available > Fix For: 1.13.0, 1.12.2 > > > Using the batch state backend requires that the inputs are sorted by a key. > Right now when sorting, we sort either all of the inputs or none. It is > because we expect that the same key is consumed from all inputs at the same > time. > In the process of implementing FLINK-20491 we can also add support for mixed > non-keyed/keyed operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20491) Support Broadcast Operation in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-20491:
-
Fix Version/s: 1.12.2
> Support Broadcast Operation in BATCH execution mode
> ---
>
> Key: FLINK-20491
> URL: https://issues.apache.org/jira/browse/FLINK-20491
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.2
>
>
> Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
> {{BATCH}} execution mode. I believe we can add support for this with not too
> much work.
> The key insight is that we can process the broadcast side before the
> non-broadcast side. Initially, we were shying away from this because of
> concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
> the user function to access/iterate over state from the keyed side. We
> thought that we couldn't support this. However, since we know that we process
> the broadcast side first we know that the keyed side will always be empty
> when doing so. We can thus just make this "keyed iteration" call a no-op,
> instead of throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20491) Support Broadcast Operation in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267326#comment-17267326
]
Aljoscha Krettek commented on FLINK-20491:
--
release-1.12: 32f07395a86554ceb079a34f0c96356b73e67b99
and previous commits
> Support Broadcast Operation in BATCH execution mode
> ---
>
> Key: FLINK-20491
> URL: https://issues.apache.org/jira/browse/FLINK-20491
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.2
>
>
> Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
> {{BATCH}} execution mode. I believe we can add support for this with not too
> much work.
> The key insight is that we can process the broadcast side before the
> non-broadcast side. Initially, we were shying away from this because of
> concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
> the user function to access/iterate over state from the keyed side. We
> thought that we couldn't support this. However, since we know that we process
> the broadcast side first we know that the keyed side will always be empty
> when doing so. We can thus just make this "keyed iteration" call a no-op,
> instead of throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17011) Introduce builder to create AbstractStreamOperatorTestHarness for testing
[
https://issues.apache.org/jira/browse/FLINK-17011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265875#comment-17265875
]
Aljoscha Krettek commented on FLINK-17011:
--
I think introducing a builder pattern here is good. [~yunta], feel free to take
over the PR and update it.
> Introduce builder to create AbstractStreamOperatorTestHarness for testing
> -
>
> Key: FLINK-17011
> URL: https://issues.apache.org/jira/browse/FLINK-17011
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, Tests
>Affects Versions: 1.10.0
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Current \{{AbstractStreamOperatorTestHarness}} lacks of builder which leads
> us to create more constructors. Moreover, to set customized component, we
> might have to call \{{AbstractStreamOperatorTestHarness#setup}}, which might
> be treated a deprecated interface, before using.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20969) Update Scala 2.12 version to 2.12.13
[ https://issues.apache.org/jira/browse/FLINK-20969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17264743#comment-17264743 ] Aljoscha Krettek commented on FLINK-20969: -- [~eloisant] Please continue on FLINK-12461. This issue is a bit more complicated, sorry for that. > Update Scala 2.12 version to 2.12.13 > > > Key: FLINK-20969 > URL: https://issues.apache.org/jira/browse/FLINK-20969 > Project: Flink > Issue Type: Task > Components: API / Scala >Reporter: Erwan Loisant >Priority: Major > Labels: pull-request-available > > Flink is compiled with Scala 2.12.7, that was released in 2018. > The latest version of the 2.12 series is 2.12.13, and upgrading will bring > build performance improvements as well as improving compatibility for > projects using more recent versions of Scala. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20959) How to close Apache Flink REST API
[ https://issues.apache.org/jira/browse/FLINK-20959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20959: - Component/s: (was: API / Core) Runtime / REST > How to close Apache Flink REST API > -- > > Key: FLINK-20959 > URL: https://issues.apache.org/jira/browse/FLINK-20959 > Project: Flink > Issue Type: Bug > Components: Runtime / REST >Affects Versions: 1.10.2 >Reporter: wuchangwen >Priority: Major > Fix For: 1.10.2 > > > Apache Flink 1.10.2 has CVE-2020-17518 vulnerability in the REST API. Now > that I want to turn off the REST API service, how should I set up the > configuration file? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20888) ContinuousFileReaderOperator should not close the output on close()
[
https://issues.apache.org/jira/browse/FLINK-20888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260480#comment-17260480
]
Aljoscha Krettek commented on FLINK-20888:
--
Nice debugging! I think that in general operators are never supposed to call
{{close()}} on their outputs.
> ContinuousFileReaderOperator should not close the output on close()
> ---
>
> Key: FLINK-20888
> URL: https://issues.apache.org/jira/browse/FLINK-20888
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.12.0
>Reporter: Yun Gao
>Priority: Major
>
> Currently ContinuousFileReaderOperator would close the output on close(), if
> it is chained with more operators, it would also close the following
> operators before their endOfInput() is called (by default, we would call
> op1.endInput(), op1.close(), op2.endInput(), op2.close()... in order). This
> might cause some problems like in
> [https://lists.apache.org/thread.html/r50a94aaea4fe25f3927a4274ea8272e6b76ecec8f3fe48d2566689bd%40%3Cuser.flink.apache.org%3E]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20491) Support Broadcast Operation in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-20491.
Fix Version/s: 1.13.0
Resolution: Implemented
master: 606c44bf2888bbe80c4eb54cac44204ae9e3198e
and previous commits
> Support Broadcast Operation in BATCH execution mode
> ---
>
> Key: FLINK-20491
> URL: https://issues.apache.org/jira/browse/FLINK-20491
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
> {{BATCH}} execution mode. I believe we can add support for this with not too
> much work.
> The key insight is that we can process the broadcast side before the
> non-broadcast side. Initially, we were shying away from this because of
> concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
> the user function to access/iterate over state from the keyed side. We
> thought that we couldn't support this. However, since we know that we process
> the broadcast side first we know that the keyed side will always be empty
> when doing so. We can thus just make this "keyed iteration" call a no-op,
> instead of throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20517) Support mixed keyed/non-keyed operations in BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-20517. Fix Version/s: (was: 1.12.1) 1.13.0 Resolution: Implemented master: 9bec3359d5b4f97fada96fd1c00c7d9cae50866d > Support mixed keyed/non-keyed operations in BATCH execution mode > > > Key: FLINK-20517 > URL: https://issues.apache.org/jira/browse/FLINK-20517 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Dawid Wysakowicz >Assignee: Aljoscha Krettek >Priority: Critical > Labels: pull-request-available > Fix For: 1.13.0 > > > Using the batch state backend requires that the inputs are sorted by a key. > Right now when sorting, we sort either all of the inputs or none. It is > because we expect that the same key is consumed from all inputs at the same > time. > In the process of implementing FLINK-20491 we can also add support for mixed > non-keyed/keyed operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17260342#comment-17260342
]
Aljoscha Krettek commented on FLINK-20870:
--
Could you maybe show the program code you're using? I think this is probably
the expected behaviour, though. There is nothing in Flink that will guarantee
that all the messages are written in one transaction.
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20870) FlinkKafkaSink
[
https://issues.apache.org/jira/browse/FLINK-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-20870:
-
Component/s: (was: API / DataStream)
Connectors / Kafka
> FlinkKafkaSink
> --
>
> Key: FLINK-20870
> URL: https://issues.apache.org/jira/browse/FLINK-20870
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
> Environment: flink :1.12.0
> kafka 2.2.1
>Reporter: xx chai
>Priority: Major
>
> I consum from kafka sink to kafka,Then I split the message into then pieces.I
> guess the ten message should in one transaction. When the fifth message is
> sink kafka ,I throw a exception.but the first four are already in kafka.
> I set some parameters :
>properties.setProperty("transactional.id", "cxx");
> properties.setProperty("ack", "all");
> properties.put("enable.idempotence",true);
> properties.put("max.in.flight.requests.per.connection",5);
> properties.put("retries", 2);
> properties.setProperty("client.id", "producer-syn-2");
> properties.put("isolation.level","read_committed");
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19589) Expose S3 options for tagging and object lifecycle policy for FileSystem
[
https://issues.apache.org/jira/browse/FLINK-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259803#comment-17259803
]
Aljoscha Krettek commented on FLINK-19589:
--
I don't really have an opinion on this. As long as it works for the use case
and as long as we don't mess up the API for filesystems that don't use this I'm
fine with what you come up with.
> Expose S3 options for tagging and object lifecycle policy for FileSystem
>
>
> Key: FLINK-19589
> URL: https://issues.apache.org/jira/browse/FLINK-19589
> Project: Flink
> Issue Type: Improvement
> Components: FileSystems
>Affects Versions: 1.12.0
>Reporter: Padarn Wilson
>Assignee: Padarn Wilson
>Priority: Minor
>
> This ticket proposes to expose the management of two properties related S3
> Object management:
> - [Lifecycle configuration
> |https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html]
> - [Object
> tagging|https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.htm]
> Being able to control these is useful for people who want to manage jobs
> using S3 for checkpointing or job output, but need to control per job level
> configuration of the tagging/lifecycle for the purposes of auditing or cost
> control (for example deleting old state from S3)
> Ideally, it would be possible to control this on each object being written by
> Flink, or at least at a job level.
> _Note_*:* Some related existing properties can be set using the hadoop module
> using system properties: see for example
> {code:java}
> fs.s3a.acl.default{code}
> which sets the default ACL on written objects.
> *Solutions*:
> 1) Modify hadoop module:
> The above-linked module could be updated in order to have a new property (and
> similar for lifecycle)
> fs.s3a.tags.default
> which could be a comma separated list of tags to set. For example
> {code:java}
> fs.s3a.acl.default = "jobname:JOBNAME,owner:OWNER"{code}
> This seems like a natural place to put this logic (and is outside of Flink if
> we decide to go this way. However it does not allow for a sink and checkpoint
> to have different values for these.
> 2) Expose withTagging from module
> The hadoop module used by Flink's existing filesystem has already exposed put
> request level tagging (see
> [this|https://github.com/aws/aws-sdk-java/blob/c06822732612d7208927d2a678073098522085c3/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/PutObjectRequest.java#L292]).
> This could be used in the Flink filesystem plugin to expose these options. A
> possible approach could be to somehow incorporate it into the file path, e.g.,
> {code:java}
> path = "TAGS:s3://bucket/path"{code}
> Or possible as an option that can be applied to the checkpoint and sink
> configurations, e.g.,
> {code:java}
> env.getCheckpointingConfig().setS3Tags(TAGS) {code}
> and similar for a file sink.
> _Note_: The lifecycle can also be managed using the module: see
> [here|https://docs.aws.amazon.com/AmazonS3/latest/dev/manage-lifecycle-using-java.html].
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13414) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259550#comment-17259550 ] Aljoscha Krettek commented on FLINK-13414: -- For sure, banning our internal Scala use into its own classloader would be a yuge benefit. Still, most of the points that [~NickBurkard] mentioned would have to be done regardless of that. > Add support for Scala 2.13 > -- > > Key: FLINK-13414 > URL: https://issues.apache.org/jira/browse/FLINK-13414 > Project: Flink > Issue Type: New Feature > Components: API / Scala >Reporter: Chaoran Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13414) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258789#comment-17258789 ] Aljoscha Krettek commented on FLINK-13414: -- [~NickBurkard] or [~bobanahalf]: if you're keen you could also try and take over FLINK-10911 to try and finally make the Scala Shell work on Scala 2.12+. I hope [~zjffdu] wouldn't mind, it seemed like he didn't have time to work on this anymore, and maybe a fresh set of eyes helps with this one. > Add support for Scala 2.13 > -- > > Key: FLINK-13414 > URL: https://issues.apache.org/jira/browse/FLINK-13414 > Project: Flink > Issue Type: New Feature > Components: API / Scala >Reporter: Chaoran Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20309) UnalignedCheckpointTestBase.execute is failed
[
https://issues.apache.org/jira/browse/FLINK-20309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258785#comment-17258785
]
Aljoscha Krettek commented on FLINK-20309:
--
Another one:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11576=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=5175
> UnalignedCheckpointTestBase.execute is failed
> -
>
> Key: FLINK-20309
> URL: https://issues.apache.org/jira/browse/FLINK-20309
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Jark Wu
>Assignee: Arvid Heise
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0, 1.13.0
>
>
> https://dev.azure.com/imjark/Flink/_build/results?buildId=345=logs=0a15d512-44ac-5ba5-97ab-13a5d066c22c=634cd701-c189-5dff-24cb-606ed884db87
> {code}
> [ERROR] Failures:
> [ERROR]
> org.apache.flink.test.checkpointing.UnalignedCheckpointITCase.execute[Parallel
> cogroup, p =
> 10](org.apache.flink.test.checkpointing.UnalignedCheckpointITCase)
> [ERROR] Run 1:
> UnalignedCheckpointITCase.execute:153->UnalignedCheckpointTestBase.execute:123
> NUM_OUT_OF_ORDER
> Expected: <0L>
> but: was <10063L>
> [ERROR] Run 2:
> UnalignedCheckpointITCase.execute:153->UnalignedCheckpointTestBase.execute:125
> NUM_LOST
> Expected: <0L>
> but: was <10037L>
> [INFO]
> [INFO]
> [ERROR] Tests run: 1592, Failures: 1, Errors: 0, Skipped: 44
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20845) Drop support for Scala 2.11
[ https://issues.apache.org/jira/browse/FLINK-20845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-20845: Assignee: Nick Burkard > Drop support for Scala 2.11 > --- > > Key: FLINK-20845 > URL: https://issues.apache.org/jira/browse/FLINK-20845 > Project: Flink > Issue Type: Sub-task > Components: API / Scala >Reporter: Nick Burkard >Assignee: Nick Burkard >Priority: Major > > The first step to adding support for Scala 2.13 is to drop Scala 2.11. > Community discussion can be found > [here|https://lists.apache.org/thread.html/ra817c5b54e3de48d80e5b4e0ae67941d387ee25cf9779f5ae37d0486%40%3Cdev.flink.apache.org%3E]. > * Scala 2.11 was released in November 2017 and is quite old now. Most > open-source libraries no longer build for it. > * Upgrading libraries to support 2.13 will be much easier without 2.11. Many > do not support 2.11, 2.12 and 2.13 at the same time, so this is basically > required to get 2.13 support. > Considerations: > * The Flink Scala Shell submodule still does not support Scala 2.12. It isn't > a strict dependency for dropping Scala 2.11, but would be nice to have before > making the cut. > * Stateful functions previously needed Scala 2.11, but it looks like it [now > supports 2.12|https://github.com/apache/flink-statefun/pull/149]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20517) Support mixed keyed/non-keyed operations in BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-20517: Assignee: Aljoscha Krettek (was: Dawid Wysakowicz) > Support mixed keyed/non-keyed operations in BATCH execution mode > > > Key: FLINK-20517 > URL: https://issues.apache.org/jira/browse/FLINK-20517 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Dawid Wysakowicz >Assignee: Aljoscha Krettek >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.1 > > > Using the batch state backend requires that the inputs are sorted by a key. > Right now when sorting, we sort either all of the inputs or none. It is > because we expect that the same key is consumed from all inputs at the same > time. > In the process of implementing FLINK-20491 we can also add support for mixed > non-keyed/keyed operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20517) Support mixed keyed/non-keyed operations in BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20517: - Description: Using the batch state backend requires that the inputs are sorted by a key. Right now when sorting, we sort either all of the inputs or none. It is because we expect that the same key is consumed from all inputs at the same time. In the process of implementing FLINK-20491 we can also add support for mixed non-keyed/keyed operators. was: Using the batch state backend requires that the inputs are sorted by a key. Right now when sorting, we sort either all of the inputs or none. It is because we expect that the same key is consumed from all inputs at the same time. We should throw an exception if such a combination is used. > Support mixed keyed/non-keyed operations in BATCH execution mode > > > Key: FLINK-20517 > URL: https://issues.apache.org/jira/browse/FLINK-20517 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.1 > > > Using the batch state backend requires that the inputs are sorted by a key. > Right now when sorting, we sort either all of the inputs or none. It is > because we expect that the same key is consumed from all inputs at the same > time. > In the process of implementing FLINK-20491 we can also add support for mixed > non-keyed/keyed operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20517) Support mixed keyed/non-keyed operations in BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20517: - Summary: Support mixed keyed/non-keyed operations in BATCH execution mode (was: Disallow operators with mixed key/non-keyed inputs with BATCH state backend) > Support mixed keyed/non-keyed operations in BATCH execution mode > > > Key: FLINK-20517 > URL: https://issues.apache.org/jira/browse/FLINK-20517 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.1 > > > Using the batch state backend requires that the inputs are sorted by a key. > Right now when sorting, we sort either all of the inputs or none. It is > because we expect that the same key is consumed from all inputs at the same > time. > We should throw an exception if such a combination is used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20517) Support mixed keyed/non-keyed operations in BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20517: - Issue Type: Improvement (was: Bug) > Support mixed keyed/non-keyed operations in BATCH execution mode > > > Key: FLINK-20517 > URL: https://issues.apache.org/jira/browse/FLINK-20517 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.1 > > > Using the batch state backend requires that the inputs are sorted by a key. > Right now when sorting, we sort either all of the inputs or none. It is > because we expect that the same key is consumed from all inputs at the same > time. > We should throw an exception if such a combination is used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16764) Allow custom Kafka topic discovery logic
[ https://issues.apache.org/jira/browse/FLINK-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258193#comment-17258193 ] Aljoscha Krettek commented on FLINK-16764: -- I'm afraid no-one worked on this so far. Do you have specific changes in mind? > Allow custom Kafka topic discovery logic > > > Key: FLINK-16764 > URL: https://issues.apache.org/jira/browse/FLINK-16764 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.10.0 >Reporter: pine zhao >Priority: Major > > Flink Kafka Consumer is also capable of discovering topics, based on pattern > matching on the topic names using regular expressions。but i need customize > discovering strategy,now it's not work,so i have to download the source code > of connectors and change them 。We need customize KafkaTopicsDescriptor -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20843) UnalignedCheckpointITCase is unstable
Aljoscha Krettek created FLINK-20843: Summary: UnalignedCheckpointITCase is unstable Key: FLINK-20843 URL: https://issues.apache.org/jira/browse/FLINK-20843 Project: Flink Issue Type: Bug Components: Runtime / Network Reporter: Aljoscha Krettek https://dev.azure.com/aljoschakrettek/Flink/_build/results?buildId=493=logs=6e55a443-5252-5db5-c632-109baf464772=9df6efca-61d0-513a-97ad-edb76d85786a=9432 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-13414) Add support for Scala 2.13
[ https://issues.apache.org/jira/browse/FLINK-13414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258133#comment-17258133 ] Aljoscha Krettek commented on FLINK-13414: -- For reference: - Thread about dropping Scala 2.11, there seems to be mostly consensus: https://lists.apache.org/thread.html/ra817c5b54e3de48d80e5b4e0ae67941d387ee25cf9779f5ae37d0486%40%3Cdev.flink.apache.org%3E - Scala Shell doesn't work with Scala 2.12: https://issues.apache.org/jira/browse/FLINK-10911 I'd be very happy if someone would take more ownership of our Scala support! 珞 We don't have any committers that really see that part as their main responsibility. I think the plan you outlined makes sense. Obviously, we would have to start by dropping 2.11. I see that as a prerequisite to all the other work. And it would be good if we could avoid breaking existing Scala 2.12 Flink jobs, but I gather your proposals would not do that. cc [~sjwiesman] > Add support for Scala 2.13 > -- > > Key: FLINK-13414 > URL: https://issues.apache.org/jira/browse/FLINK-13414 > Project: Flink > Issue Type: New Feature > Components: API / Scala >Reporter: Chaoran Yu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255509#comment-17255509 ] Aljoscha Krettek commented on FLINK-20628: -- Ok, that sounds good! > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Major > Fix For: 1.12.0 > > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20625) Refactor Google Cloud PubSub Source in accordance with FLIP-27
[ https://issues.apache.org/jira/browse/FLINK-20625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255508#comment-17255508 ] Aljoscha Krettek commented on FLINK-20625: -- Thanks, that looks good! And yes, we don't need to solve everything in one go. > Refactor Google Cloud PubSub Source in accordance with FLIP-27 > -- > > Key: FLINK-20625 > URL: https://issues.apache.org/jira/browse/FLINK-20625 > Project: Flink > Issue Type: Improvement > Components: Connectors / Google Cloud PubSub >Reporter: Jakob Edding >Priority: Major > > The Source implementation for Google Cloud Pub/Sub should be refactored in > accordance with [FLIP-27: Refactor Source > Interface|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95653748]. > *Split Enumerator* > Pub/Sub doesn't expose any partitions to consuming applications. Therefore, > the implementation of the Pub/Sub Split Enumerator won't do any real work > discovery. Instead, a static Source Split is handed to Source Readers which > request a Source Split. This static Source Split merely contains details > about the connection to Pub/Sub and the concrete Pub/Sub subscription to use > but no Split-specific information like partitions/offsets because this > information can not be obtained. > *Source Reader* > A Source Reader will use Pub/Sub's pull mechanism to read new messages from > the Pub/Sub subscription specified in the SourceSplit. In the case of > parallel-running Source Readers in Flink, every Source Reader will be passed > the same Source Split from the Enumerator. Because of this, all Source > Readers use the same connection details and the same Pub/Sub subscription to > receive messages. In this case, Pub/Sub will automatically load-balance > messages between all Source Readers pulling from the same subscription. This > way, parallel processing can be achieved in the Source. > *At-least-once guarantee* > Pub/Sub itself guarantees at-least-once message delivery so it is the goal to > keep up this guarantee in the Source as well. A mechanism that can be used to > achieve this is that Pub/Sub expects a message to be acknowledged by the > subscriber to signal that the message has been consumed successfully. Any > message that has not been acknowledged yet will be automatically redelivered > by Pub/Sub once an ack deadline has passed. > After a certain time interval has elapsed... > # all pulled messages are checkpointed in the Source Reader > # same messages are acknowledged to Pub/Sub > # same messages are forwarded to downstream Flink tasks > This should ensure at-least-once delivery in the Source because in the case > of failure, non-checkpointed messages have not yet been acknowledged and will > therefore be redelivered by the Pub/Sub service. > Because of the static Source Split, it appears like checkpointing is not > necessary in the Split Enumerator. > *Possible exactly-once guarantee* > It should even be possible to achieve exactly-once guarantees for the source. > The following requirements would have to be met to have an exactly-once mode > besides the at-least-once mode similar to how it is done in the [current > RabbitMQ > Source|https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/rabbitmq.html#rabbitmq-source]: > * The system which publishes messages to Pub/Sub must add an id to each > message so that messages can be deduplicated in the Source. > * The Source must run in a non-parallel fashion (with parallelism=1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17691) FlinkKafkaProducer transactional.id too long when using Semantic.EXACTLY_ONCE
[
https://issues.apache.org/jira/browse/FLINK-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255506#comment-17255506
]
Aljoscha Krettek commented on FLINK-17691:
--
Regarding Issues #1 and #2, these were proposed as potential fixes for this
problem but in the end we went with the simplest solution, which is to just
truncate the name. The Kafka transactional id and the operator UIDs that are
mentioned in the documentation are not related, the latter is used only for
mapping state to operators.
Regarding your issues #3 and #4, I think you're right. We could make the
transactional Id *prefix* configurable, it's just that not enough users have
asked for it so far. Or anyone really.
> FlinkKafkaProducer transactional.id too long when using Semantic.EXACTLY_ONCE
> -
>
> Key: FLINK-17691
> URL: https://issues.apache.org/jira/browse/FLINK-17691
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0, 1.11.0
>Reporter: freezhan
>Assignee: John Mathews
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments: image-2020-05-14-20-43-57-414.png,
> image-2020-05-14-20-45-24-030.png, image-2020-05-14-20-45-59-878.png,
> image-2020-05-14-21-09-01-906.png, image-2020-05-14-21-16-43-810.png,
> image-2020-05-14-21-17-09-784.png
>
>
> When sink to Kafka using the {color:#FF}Semantic.EXACTLY_ONCE {color}mode.
> The flink Kafka Connector Producer will auto set the
> {color:#FF}transactional.id{color}, and the user - defined value are
> ignored.
>
> When the job operator name too long, will send failed
> transactional.id is exceeds the kafka {color:#FF}coordinator_key{color}
> limit
> !image-2020-05-14-21-09-01-906.png!
>
> *The flink Kafka Connector policy for automatic generation of transaction.id
> is as follows*
>
> 1. use the {color:#FF}taskName + "-" + operatorUniqueID{color} as
> transactional.id prefix (may be too long)
> getRuntimeContext().getTaskName() + "-" + ((StreamingRuntimeContext)
> getRuntimeContext()).getOperatorUniqueID()
> 2. Range of available transactional ids
> [nextFreeTransactionalId, nextFreeTransactionalId + parallelism *
> kafkaProducersPoolSize)
> !image-2020-05-14-20-43-57-414.png!
> !image-2020-05-14-20-45-24-030.png!
> !image-2020-05-14-20-45-59-878.png!
>
> *The Kafka transaction.id check policy as follows:*
>
> {color:#FF}string bytes.length can't larger than Short.MAX_VALUE
> (32767){color}
> !image-2020-05-14-21-16-43-810.png!
> !image-2020-05-14-21-17-09-784.png!
>
> *To reproduce this bug, the following conditions must be met:*
>
> # send msg to kafka with exactly once mode
> # the task TaskName' length + TaskName's length is lager than the 32767 (A
> very long line of SQL or window statements can appear)
> *I suggest a solution:*
>
> 1. Allows users to customize transactional.id 's prefix
> or
> 2. Do md5 on the prefix before returning the real transactional.id
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20764) BatchGroupedReduceOperator does not emit results for singleton inputs
[ https://issues.apache.org/jira/browse/FLINK-20764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20764: - Issue Type: Bug (was: Improvement) > BatchGroupedReduceOperator does not emit results for singleton inputs > - > > Key: FLINK-20764 > URL: https://issues.apache.org/jira/browse/FLINK-20764 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.1 > > > This was reported on the ML: > https://lists.apache.org/thread.html/rf24ff56408d3f07a02a3726336229a5919e62c4179c9eb8361a5b76b%40%3Cuser.flink.apache.org%3E. > The problem is that the logic for setting up the "final" timer does not kick > in when there is only on input. > I'll cut a PR for that and also add tests for the operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20764) BatchGroupedReduceOperator does not emit results for singleton inputs
Aljoscha Krettek created FLINK-20764: Summary: BatchGroupedReduceOperator does not emit results for singleton inputs Key: FLINK-20764 URL: https://issues.apache.org/jira/browse/FLINK-20764 Project: Flink Issue Type: Improvement Components: API / DataStream Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Fix For: 1.12.1 This was reported on the ML: https://lists.apache.org/thread.html/rf24ff56408d3f07a02a3726336229a5919e62c4179c9eb8361a5b76b%40%3Cuser.flink.apache.org%3E. The problem is that the logic for setting up the "final" timer does not kick in when there is only on input. I'll cut a PR for that and also add tests for the operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20476) New File Sink end-to-end test Failed
[
https://issues.apache.org/jira/browse/FLINK-20476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-20476.
Resolution: Not A Problem
Closing this for now since it wasn't actually a problem with the {{FileSink}}
and that might be misleading. Please re-open if it does happen again.
> New File Sink end-to-end test Failed
>
>
> Key: FLINK-20476
> URL: https://issues.apache.org/jira/browse/FLINK-20476
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0, 1.12.1
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10502=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529]
> {code}
> 2020-12-03T23:22:43.8578352Z Dec 03 23:22:43 Starting taskexecutor daemon on
> host fv-az586-109.
> 2020-12-03T23:22:43.8587276Z Dec 03 23:22:43 Waiting for restart to happen
> 2020-12-03T23:22:43.8587669Z Dec 03 23:22:43 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:22:48.9939434Z Dec 03 23:22:48 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:22:54.1236439Z Dec 03 23:22:54 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:22:59.2469617Z Dec 03 23:22:59 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:04.3730041Z Dec 03 23:23:04 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:09.5227739Z Dec 03 23:23:09 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:14.6572986Z Dec 03 23:23:14 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:19.7762483Z Dec 03 23:23:19 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:24.8973187Z Dec 03 23:23:24 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:30.0272934Z Dec 03 23:23:30 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:35.2332771Z Dec 03 23:23:35 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:40.3766421Z Dec 03 23:23:40 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:45.5103677Z Dec 03 23:23:45 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:50.6382894Z Dec 03 23:23:50 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:55.7908088Z Dec 03 23:23:55 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:00.9276393Z Dec 03 23:24:00 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:06.0966785Z Dec 03 23:24:06 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:11.2497761Z Dec 03 23:24:11 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:16.4118742Z Dec 03 23:24:16 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:21.5640591Z Dec 03 23:24:21 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:26.7080816Z Dec 03 23:24:26 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:31.8460471Z Dec 03 23:24:31 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:36.9640393Z Dec 03 23:24:36 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:42.1055030Z Dec 03 23:24:42 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:47.2399707Z Dec 03 23:24:47 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:52.3555612Z Dec 03 23:24:52 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:57.4903920Z Dec 03 23:24:57 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:02.6275471Z Dec 03 23:25:02 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:07.7481675Z Dec 03 23:25:07 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:12.9002926Z Dec 03 23:25:12 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:18.0361767Z Dec 03 23:25:18 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:23.1776688Z Dec 03 23:25:23 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:28.3029352Z Dec 03 23:25:28 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:33.4175706Z Dec 03 23:25:33 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:38.5444337Z Dec 03 23:25:38 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:43.6699914Z Dec 03 23:25:43 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:48.8064066Z Dec 03 23:25:48 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:53.9376640Z Dec 03
[jira] [Created] (FLINK-20651) Use Spotless/google-java-format for code formatting/enforcement
Aljoscha Krettek created FLINK-20651: Summary: Use Spotless/google-java-format for code formatting/enforcement Key: FLINK-20651 URL: https://issues.apache.org/jira/browse/FLINK-20651 Project: Flink Issue Type: Improvement Components: Build System Reporter: Aljoscha Krettek Assignee: Aljoscha Krettek Please see the ML discussion for background: https://lists.apache.org/thread.html/rfb079ec4cfd35bcb93df9c2163aaa121e392282f0f3d9710c8ade811%40%3Cdev.flink.apache.org%3E. There was broad consensus in the discussion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20476) New File Sink end-to-end test Failed
[
https://issues.apache.org/jira/browse/FLINK-20476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250968#comment-17250968
]
Aljoscha Krettek commented on FLINK-20476:
--
Something must have changed if there is suddenly this burst of failures. The
test was rather quiet since it was introduced, I think.
> New File Sink end-to-end test Failed
>
>
> Key: FLINK-20476
> URL: https://issues.apache.org/jira/browse/FLINK-20476
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.13.0, 1.12.1
>Reporter: Huang Xingbo
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.13.0, 1.12.1
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10502=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529]
> {code}
> 2020-12-03T23:22:43.8578352Z Dec 03 23:22:43 Starting taskexecutor daemon on
> host fv-az586-109.
> 2020-12-03T23:22:43.8587276Z Dec 03 23:22:43 Waiting for restart to happen
> 2020-12-03T23:22:43.8587669Z Dec 03 23:22:43 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:22:48.9939434Z Dec 03 23:22:48 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:22:54.1236439Z Dec 03 23:22:54 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:22:59.2469617Z Dec 03 23:22:59 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:04.3730041Z Dec 03 23:23:04 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:09.5227739Z Dec 03 23:23:09 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:14.6572986Z Dec 03 23:23:14 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:19.7762483Z Dec 03 23:23:19 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:24.8973187Z Dec 03 23:23:24 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:30.0272934Z Dec 03 23:23:30 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:35.2332771Z Dec 03 23:23:35 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:40.3766421Z Dec 03 23:23:40 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:45.5103677Z Dec 03 23:23:45 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:50.6382894Z Dec 03 23:23:50 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:23:55.7908088Z Dec 03 23:23:55 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:00.9276393Z Dec 03 23:24:00 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:06.0966785Z Dec 03 23:24:06 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:11.2497761Z Dec 03 23:24:11 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:16.4118742Z Dec 03 23:24:16 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:21.5640591Z Dec 03 23:24:21 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:26.7080816Z Dec 03 23:24:26 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:31.8460471Z Dec 03 23:24:31 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:36.9640393Z Dec 03 23:24:36 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:42.1055030Z Dec 03 23:24:42 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:47.2399707Z Dec 03 23:24:47 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:52.3555612Z Dec 03 23:24:52 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:24:57.4903920Z Dec 03 23:24:57 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:02.6275471Z Dec 03 23:25:02 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:07.7481675Z Dec 03 23:25:07 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:12.9002926Z Dec 03 23:25:12 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:18.0361767Z Dec 03 23:25:18 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:23.1776688Z Dec 03 23:25:23 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:28.3029352Z Dec 03 23:25:28 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:33.4175706Z Dec 03 23:25:33 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:38.5444337Z Dec 03 23:25:38 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:43.6699914Z Dec 03 23:25:43 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:48.8064066Z Dec 03 23:25:48 Still waiting for restarts.
> Expected: 1 Current: 0
> 2020-12-03T23:25:53.9376640Z Dec 03 23:25:53
[jira] [Commented] (FLINK-20646) ReduceTransformation does not work with RocksDBStateBackend
[
https://issues.apache.org/jira/browse/FLINK-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250953#comment-17250953
]
Aljoscha Krettek commented on FLINK-20646:
--
I think the windowless {{reduce()}} is not a common operation because it's not
super useful in {{STREAMING}} mode. You just get an infinite stream of updates.
Do you know how they discovered the bug?
> ReduceTransformation does not work with RocksDBStateBackend
> ---
>
> Key: FLINK-20646
> URL: https://issues.apache.org/jira/browse/FLINK-20646
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Xintong Song
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.1
>
>
> The intra-slot managed memory sharing (FLIP-141) requires transformations to
> properly declare their managed memory use cases.
> For RocksDB state backend, it requires all {{Transformation}}-s on a keyed
> stream (with non-null {{KeySelector}}) to call
> {{Transformation#updateManagedMemoryStateBackendUseCase}}, which the newly
> introduced {{ReduceTransformation}} did not.
> As a result, Flink will not reserve managed memory for operators converted
> from {{ReduceTransformation}} (FLINK-19931), leading to the following failure
> when RocksDB state backend is used.
> {code}
> 16:58:49,373 WARN
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure [] -
> Exception while restoring keyed state backend for
> StreamGroupedReduceOperator_c27dcf7b54ef6bfd6cff02ca8870b681_(1/1) from
> alternative (1/1), will retry while more alternatives are available.
> java.io.IOException: Failed to acquire shared cache resource for RocksDB
> at
> org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.allocateSharedCachesIfConfigured(RocksDBOperationUtils.java:264)
> ~[flink-statebackend-rocksdb_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:535)
> ~[flink-statebackend-rocksdb_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:94)
> ~[flink-statebackend-rocksdb_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:299)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:316)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:155)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:248)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:400)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:507)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> [flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> [flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:531)
> [flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> [flink-runtime_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> [flink-runtime_2.11-1.12.0.jar:1.12.0]
> at java.lang.Thread.run(Thread.java:832) [?:?]
> Caused by: java.lang.IllegalArgumentException: The fraction of memory to
> allocate should not be 0. Please make sure that all types of managed memory
> consumers contained in the job are configured with a non-negative weight via
[jira] [Commented] (FLINK-20646) ReduceTransformation does not work with RocksDBStateBackend
[
https://issues.apache.org/jira/browse/FLINK-20646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250942#comment-17250942
]
Aljoscha Krettek commented on FLINK-20646:
--
A very quick fix for this is to just call
{code}
updateManagedMemoryStateBackendUseCase(true);
{code}
in {{ReduceTransformation}}. I wouldn't add a test for it now and it seems we
also don't have equivalent tests for other transformations.
One thing I noticed is that there are some conflicts in how we assign the
memory requirements. We allow operators/transformations to do it but we also
hard-set some things for batch execution mode. This comes from the fact that
the Blink-derived Table runner basically re-implements the newer
BATCH-execution machinery we now have for general operators/transformations. In
the long run, we should get rid of this complication.
> ReduceTransformation does not work with RocksDBStateBackend
> ---
>
> Key: FLINK-20646
> URL: https://issues.apache.org/jira/browse/FLINK-20646
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Xintong Song
>Assignee: Xintong Song
>Priority: Blocker
> Fix For: 1.12.1
>
>
> The intra-slot managed memory sharing (FLIP-141) requires transformations to
> properly declare their managed memory use cases.
> For RocksDB state backend, it requires all {{Transformation}}-s on a keyed
> stream (with non-null {{KeySelector}}) to call
> {{Transformation#updateManagedMemoryStateBackendUseCase}}, which the newly
> introduced {{ReduceTransformation}} did not.
> As a result, Flink will not reserve managed memory for operators converted
> from {{ReduceTransformation}} (FLINK-19931), leading to the following failure
> when RocksDB state backend is used.
> {code}
> 16:58:49,373 WARN
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure [] -
> Exception while restoring keyed state backend for
> StreamGroupedReduceOperator_c27dcf7b54ef6bfd6cff02ca8870b681_(1/1) from
> alternative (1/1), will retry while more alternatives are available.
> java.io.IOException: Failed to acquire shared cache resource for RocksDB
> at
> org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.allocateSharedCachesIfConfigured(RocksDBOperationUtils.java:264)
> ~[flink-statebackend-rocksdb_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:535)
> ~[flink-statebackend-rocksdb_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:94)
> ~[flink-statebackend-rocksdb_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:299)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:316)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:155)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:248)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:400)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:507)
> ~[flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> [flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:501)
> [flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:531)
> [flink-streaming-java_2.11-1.12.0.jar:1.12.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
>
[jira] [Commented] (FLINK-20625) Refactor Google Cloud PubSub Source in accordance with FLIP-27
[ https://issues.apache.org/jira/browse/FLINK-20625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250898#comment-17250898 ] Aljoscha Krettek commented on FLINK-20625: -- Does our current PubSub source also only support at-least-once delivery? I'm too lazy to look into it now, to see if we maybe to custom deduplication after the source. > Refactor Google Cloud PubSub Source in accordance with FLIP-27 > -- > > Key: FLINK-20625 > URL: https://issues.apache.org/jira/browse/FLINK-20625 > Project: Flink > Issue Type: Improvement > Components: Connectors / Google Cloud PubSub >Reporter: Jakob Edding >Priority: Major > > The Source implementation for Google Cloud Pub/Sub should be refactored in > accordance with [FLIP-27: Refactor Source > Interface|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95653748]. > *Split Enumerator* > Pub/Sub doesn't expose any partitions to consuming applications. Therefore, > the implementation of the Pub/Sub Split Enumerator won't do any real work > discovery. Instead, a static Source Split is handed to Source Readers which > request a Source Split. This static Source Split merely contains details > about the connection to Pub/Sub and the concrete Pub/Sub subscription to use > but no Split-specific information like partitions/offsets because this > information can not be obtained. > *Source Reader* > A Source Reader will use Pub/Sub's pull mechanism to read new messages from > the Pub/Sub subscription specified in the SourceSplit. In the case of > parallel-running Source Readers in Flink, every Source Reader will be passed > the same Source Split from the Enumerator. Because of this, all Source > Readers use the same connection details and the same Pub/Sub subscription to > receive messages. In this case, Pub/Sub will automatically load-balance > messages between all Source Readers pulling from the same subscription. This > way, parallel processing can be achieved in the Source. > *At-least-once guarantee* > Pub/Sub itself guarantees at-least-once message delivery so it is the goal to > keep up this guarantee in the Source as well. A mechanism that can be used to > achieve this is that Pub/Sub expects a message to be acknowledged by the > subscriber to signal that the message has been consumed successfully. Any > message that has not been acknowledged yet will be automatically redelivered > by Pub/Sub once an ack deadline has passed. > After a certain time interval has elapsed... > # all pulled messages are checkpointed in the Source Reader > # same messages are acknowledged to Pub/Sub > # same messages are forwarded to downstream Flink tasks > This should ensure at-least-once delivery in the Source because in the case > of failure, non-checkpointed messages have not yet been acknowledged and will > therefore be redelivered. > Because of the static Source Split, it appears like checkpointing is not > necessary in the Split Enumerator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20628) Port RabbitMQ Sources to FLIP-27 API
[ https://issues.apache.org/jira/browse/FLINK-20628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250896#comment-17250896 ] Aljoscha Krettek commented on FLINK-20628: -- So in this case splits would not contain a "real" split because all the readers would essentially subscribe to the same logical thing? > Port RabbitMQ Sources to FLIP-27 API > > > Key: FLINK-20628 > URL: https://issues.apache.org/jira/browse/FLINK-20628 > Project: Flink > Issue Type: Improvement > Components: Connectors/ RabbitMQ >Reporter: Jan Westphal >Priority: Major > Fix For: 1.12.0 > > > *Structure* > The new RabbitMQ Source will have three components: > * RabbitMQ enumerator that receives one RabbitMQ Channel Config. > * RabbitMQ splits contain the RabbitMQ Channel Config > * RabbitMQ Readers which subscribe to the same RabbitMQ channel and receive > the messages (automatically load balanced by RabbitMQ). > *Checkpointing Enumerators* > The enumerator only needs to checkpoint the RabbitMQ channel config since the > continuous discovery of new unread/unhandled messages is taken care of by the > subscribed RabbitMQ readers and RabbitMQ itself. > *Checkpointing Readers* > The new RabbitMQ Source needs to ensure that every reader can be checkpointed. > Since RabbitMQ is non-persistent and cannot be read by offset, a combined > usage of checkpoints and message acknowledgments is necessary. Until a > received message is checkpointed by a reader, it will stay in an > un-acknowledge state. As soon as the checkpoint is created, the messages from > the last checkpoint can be acknowledged as handled against RabbitMQ and thus > will be deleted only then. Messages need to be acknowledged one by one as > messages are handled by each SourceReader individually. > When deserializing the messages we will make use of the implementation in the > existing RabbitMQ Source. > *Message Delivery Guarantees* > Unacknowledged messages of a reader will be redelivered by RabbitMQ > automatically to other consumers of the same channel if the reader goes down. > > This Source is going to only support at-least-once as this is the default > RabbitMQ behavior and thus everything else would require changes to RabbitMQ > itself or would impair the idea of parallelizing SourceReaders. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20618) Some of the source operator subtasks will stuck when flink job in critical backpressure
[ https://issues.apache.org/jira/browse/FLINK-20618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20618: - Component/s: (was: API / Core) Runtime / Network > Some of the source operator subtasks will stuck when flink job in critical > backpressure > --- > > Key: FLINK-20618 > URL: https://issues.apache.org/jira/browse/FLINK-20618 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.11.0, 1.10.2 >Reporter: zlzhang0122 >Priority: Critical > Attachments: 2020-12-16 11-47-37 的屏幕截图.png, 2020-12-16 11-48-30 > 的屏幕截图.png, 2020-12-16 11-49-01 的屏幕截图.png, 2020-12-16 11-53-42 的屏幕截图.png > > > In some critical backpressure situation, some of the subtasks of source will > blocked to request buffer because of the LocalBufferPool is full,so the whole > task will be stuck and the other task run well. > Bellow is the jstack trace: > > Legacy Source Thread - Source: TalosTableSource(hash, timestamp, step, > isCustomize, hostname, endpoint, metric, dsType, orgId, idc, labels, pdl) > - SourceConversion(table=[default_catalog.default_database.transfer_c5, > source: [TalosTableSource(hash, timestamp, step, isCustomize, hostname, > endpoint, metric, dsType, orgId, idc, labels, pdl)]], fields=[hash, > timestamp, step, isCustomize, hostname, endpoint, metric, dsType, orgId, idc, > labels, pdl]) - Calc(select=[hash, timestamp, step, isCustomize, > hostname, endpoint, metric, dsType, orgId, idc, labels, pdl, () AS $12, > (timestamp - ((timestamp + 18000) MOD 86400)) AS $13]) (62/128) #113 prio=5 > os_prio=0 tid=0x7f43d07e1800 nid=0x1b1c waiting on condition > [0x7f43b8488000] > java.lang.Thread.State: WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for 0xdb234488 (a > java.util.concurrent.CompletableFuture$Signaller) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693) > at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) > at > java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729) > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestMemorySegmentBlocking(LocalBufferPool.java:231) > at > org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:209) > at > org.apache.flink.runtime.io.network.partition.ResultPartition.getBufferBuilder(ResultPartition.java:189) > at > org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.requestNewBufferBuilder(ChannelSelectorRecordWriter.java:103) > at > org.apache.flink.runtime.io.network.api.writer.RecordWriter.copyFromSerializerToTargetChannel(RecordWriter.java:145) > at > org.apache.flink.runtime.io.network.api.writer.RecordWriter.emit(RecordWriter.java:116) > at > org.apache.flink.runtime.io.network.api.writer.ChannelSelectorRecordWriter.emit(ChannelSelectorRecordWriter.java:60) > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:107) > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89) > at > org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:732) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:710) > at StreamExecCalc$33.processElement(Unknown Source) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:616) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:596) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:732) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:710) > at SourceConversion$4.processElement(Unknown Source) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:641) > at >
[jira] [Commented] (FLINK-20617) Kafka Consumer Deserializer Exception on application mode
[
https://issues.apache.org/jira/browse/FLINK-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17250216#comment-17250216
]
Aljoscha Krettek commented on FLINK-20617:
--
I think this is because of a dependency mismatch. You're excluding
{{kafka-clients}} and including a custom version.
> Kafka Consumer Deserializer Exception on application mode
> -
>
> Key: FLINK-20617
> URL: https://issues.apache.org/jira/browse/FLINK-20617
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.11.2
> Environment: application mode
> flink 1.11.2 with hadoop 2.6.0-cdh5.15.0
>Reporter: Georger
>Priority: Critical
>
> Kafka source may has some issues on application mode
>
> when i run it with application mode on flink 1.11.2 it can't startup
> the detail Excetion is:
> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:789)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:643)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:623)
> at
> org.apache.flink.streaming.connectors.kafka.internal.KafkaPartitionDiscoverer.initializeConnections(KafkaPartitionDiscoverer.java:58)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractPartitionDiscoverer.open(AbstractPartitionDiscoverer.java:94)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.open(FlinkKafkaConsumerBase.java:550)
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:479)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:475)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:528)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.KafkaException:
> org.apache.kafka.common.serialization.ByteArrayDeserializer is not an
> instance of org.apache.kafka.common.serialization.Deserializer
> at
> org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:263)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:688)
> ... 15 more
> The pom is:
>
> org.apache.flink
> flink-connector-kafka_2.11
> ${flink.version}
>
>
> org.slf4j
> slf4j-api
>
>
> org.apache.kafka
> kafka-clients
>
>
>
>
> org.apache.kafka
> kafka-clients
> 1.0.1
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20541) ClusterID is not used in the method!
[
https://issues.apache.org/jira/browse/FLINK-20541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-20541:
-
Fix Version/s: (was: 1.11.3)
> ClusterID is not used in the method!
>
>
> Key: FLINK-20541
> URL: https://issues.apache.org/jira/browse/FLINK-20541
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission
>Affects Versions: 1.11.2
>Reporter: lixiaobao
>Priority: Minor
> Labels: pull-request-available
>
> {code:java}
> package org.apache.flink.client.cli;
> import org.apache.flink.annotation.Internal;
> import
> org.apache.flink.client.deployment.application.ApplicationConfiguration;
> import org.apache.flink.configuration.Configuration;
> /**
> * An interface to be used by the {@link CliFrontend}
> * to submit user programs for execution.
> */
> @Internal
> public interface ApplicationDeployer {
>/**
> * Submits a user program for execution and runs the main user method on
> the cluster.
> *
> * @param configuration the configuration containing all the necessary
> *information about submitting the user program.
> * @param applicationConfiguration an {@link ApplicationConfiguration}
> specific to
> * the application to be executed.
> */
> void run(
> final Configuration configuration,
> final ApplicationConfiguration applicationConfiguration) throws
> Exception;
> }
> {code}
> is not used
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20491) Support Broadcast Operation in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-20491:
-
Summary: Support Broadcast Operation in BATCH execution mode (was: Support
Broadcast State in BATCH execution mode)
> Support Broadcast Operation in BATCH execution mode
> ---
>
> Key: FLINK-20491
> URL: https://issues.apache.org/jira/browse/FLINK-20491
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
>
> Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
> {{BATCH}} execution mode. I believe we can add support for this with not too
> much work.
> The key insight is that we can process the broadcast side before the
> non-broadcast side. Initially, we were shying away from this because of
> concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
> the user function to access/iterate over state from the keyed side. We
> thought that we couldn't support this. However, since we know that we process
> the broadcast side first we know that the keyed side will always be empty
> when doing so. We can thus just make this "keyed iteration" call a no-op,
> instead of throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20491) Support Broadcast State in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247268#comment-17247268
]
Aljoscha Krettek commented on FLINK-20491:
--
I extended my work-in-progress PR: https://github.com/apache/flink/pull/14312
For the wiring, I'm extending {{StreamConfig.InputConfig}} to allow specifying
a requirement of {{SORTED}} or {{PASS_THROUGH}}. With this, I'm removing the
"global" {{SORTED_INPUTS}} field of {{StreamConfig}} because we now allow more
fine-grained control.
I still have to clean up the commits and add more Javadocs, but WDYT so far?
> Support Broadcast State in BATCH execution mode
> ---
>
> Key: FLINK-20491
> URL: https://issues.apache.org/jira/browse/FLINK-20491
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
>
> Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
> {{BATCH}} execution mode. I believe we can add support for this with not too
> much work.
> The key insight is that we can process the broadcast side before the
> non-broadcast side. Initially, we were shying away from this because of
> concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
> the user function to access/iterate over state from the keyed side. We
> thought that we couldn't support this. However, since we know that we process
> the broadcast side first we know that the keyed side will always be empty
> when doing so. We can thus just make this "keyed iteration" call a no-op,
> instead of throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20491) Support Broadcast State in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246439#comment-17246439
]
Aljoscha Krettek commented on FLINK-20491:
--
Thanks, those are very thoughtful comments! I'll try and answer individually.
I think we don't need to sort the broadcast side. My assumption is that the
broadcast side is for building up some state that the other side is then
"streamed by" (even though it's not really streamed for BATCH execution).
Ordering by time should not be required for that case. If we see that it
becomes a requirement in the future we could still do it.
The above resolves questions about sorting non-keyed inputs and the {{byte[]}}
array trick for now, by sidestepping them.
I would like to completely separate the different types of input, yes. What I
would do next is change {{MultiInputSortingDataInput.wrapInputs()}} to pass in
separately: inputs that need sorting and pass-through inputs.
This leaves the problem of the {{BatchExecutionKeyedStateBackend}}. It's true
that we also cannot support iterating over keys from the keyed side. However, I
think we need to silently ignore it because real-word programs will have to use
this from the broadcast side for {{STREAMING}} execution. I'll think about what
the implications are some more. Maybe it's fine to make it a no-op for both
sides. Or maybe we'll disallow iterating from the keyed side for all execution
modes. Although we then have to figure out how we determine "on which side" we
are.
> Support Broadcast State in BATCH execution mode
> ---
>
> Key: FLINK-20491
> URL: https://issues.apache.org/jira/browse/FLINK-20491
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
>
> Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
> {{BATCH}} execution mode. I believe we can add support for this with not too
> much work.
> The key insight is that we can process the broadcast side before the
> non-broadcast side. Initially, we were shying away from this because of
> concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
> the user function to access/iterate over state from the keyed side. We
> thought that we couldn't support this. However, since we know that we process
> the broadcast side first we know that the keyed side will always be empty
> when doing so. We can thus just make this "keyed iteration" call a no-op,
> instead of throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20491) Support Broadcast State in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246380#comment-17246380
]
Aljoscha Krettek commented on FLINK-20491:
--
I posted a first PoC as a PR. Right now, what I'm doing is to "prefer" a given
input side (this is hardcoded right now but should be passed in from outside).
Also, I'm changing the sorter to use an empty {{byte[]}} array for the input
where we don't have keys, meaning we also sort the broadcast input by timestamp.
> Support Broadcast State in BATCH execution mode
> ---
>
> Key: FLINK-20491
> URL: https://issues.apache.org/jira/browse/FLINK-20491
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
> Labels: pull-request-available
>
> Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
> {{BATCH}} execution mode. I believe we can add support for this with not too
> much work.
> The key insight is that we can process the broadcast side before the
> non-broadcast side. Initially, we were shying away from this because of
> concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
> the user function to access/iterate over state from the keyed side. We
> thought that we couldn't support this. However, since we know that we process
> the broadcast side first we know that the keyed side will always be empty
> when doing so. We can thus just make this "keyed iteration" call a no-op,
> instead of throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20491) Support Broadcast State in BATCH execution mode
Aljoscha Krettek created FLINK-20491:
Summary: Support Broadcast State in BATCH execution mode
Key: FLINK-20491
URL: https://issues.apache.org/jira/browse/FLINK-20491
Project: Flink
Issue Type: Improvement
Components: API / DataStream
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Right now, we don't support {{DataStream.connect(BroadcastStream)}} in
{{BATCH}} execution mode. I believe we can add support for this with not too
much work.
The key insight is that we can process the broadcast side before the
non-broadcast side. Initially, we were shying away from this because of
concerns about {{ctx.applyToKeyedState()}} which allows the broadcast side of
the user function to access/iterate over state from the keyed side. We thought
that we couldn't support this. However, since we know that we process the
broadcast side first we know that the keyed side will always be empty when
doing so. We can thus just make this "keyed iteration" call a no-op, instead of
throwing an exception as we do now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20098) Don't add flink-connector-files to flink-dist, make dependencies explicit
[
https://issues.apache.org/jira/browse/FLINK-20098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243207#comment-17243207
]
Aljoscha Krettek commented on FLINK-20098:
--
The final solution shouldn't have a regression. The SQL uber jar now contains
the file connector code, because it is the component that actively "uses" it.
If you remove the SQL uber jars from {{lib/}} then you won't have the file
connector code "provided" and users should include the connector in their job
jars, like for the other connectors.
> Don't add flink-connector-files to flink-dist, make dependencies explicit
> -
>
> Key: FLINK-20098
> URL: https://issues.apache.org/jira/browse/FLINK-20098
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> We currently add both {{flink-connector-files}} and {{flink-connector-base}}
> to {{flink-dist}}.
> This implies, that users should use the dependency like this:
> {code}
>
> org.apache.flink
> flink-connector-files
> ${project.version}
> provided
>
> {code}
> which differs from other connectors where users don't need to specify
> {{provided}}.
> Also, {{flink-connector-files}} has {{flink-connector-base}} as a provided
> dependency, which means that examples that use this dependency will not run
> out-of-box in IntelliJ because transitive provided dependencies will not be
> considered.
> I propose to just remove the dependencies from {{flink-dist}} and let users
> use the File Connector like any other connector.
> I believe the initial motivation for "providing" the File Connector in
> {{flink-dist}} was to allow us to use the File Connector under the hood in
> methods such as {{StreamExecutionEnvironment.readFile(...)}}. We could decide
> to deprecate and remove those methods or re-add the File Connector as an
> explicit (non-provided) dependency again in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19984) Add TypeSerializerTestCoverageTest to check whether tests based on SerializerTestBase and TypeSerializerUpgradeTestBase
[
https://issues.apache.org/jira/browse/FLINK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-19984:
-
Affects Version/s: (was: 1.12.0)
> Add TypeSerializerTestCoverageTest to check whether tests based on
> SerializerTestBase and TypeSerializerUpgradeTestBase
> ---
>
> Key: FLINK-19984
> URL: https://issues.apache.org/jira/browse/FLINK-19984
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Reporter: Nicholas Jiang
>Assignee: Nicholas Jiang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently, we have {{TypeInfoTestCoverageTest}} which checks that we have a
> test that extends {{TypeInformationTestBase}} for all type infos. But
> {{TypeSerializer}} doesn’t have the same thing that would verify that
> {{TypeSerializer}} has tests that extend {{SerializerTestBase}} and
> {{TypeSerializerUpgradeTestBase}}. Therefore we don’t know if test coverage
> of {{TypeSerializer}} is good.
> This would add {{TypeSerializerTestCoverageTest}} to check whether to have
> tests based on {{SerializerTestBase}} and {{TypeSerializerUpgradeTestBase}}
> because all serializers should have tests based on both of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19984) Add TypeSerializerTestCoverageTest to check whether tests based on SerializerTestBase and TypeSerializerUpgradeTestBase
[
https://issues.apache.org/jira/browse/FLINK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-19984:
-
Fix Version/s: 1.12.0
> Add TypeSerializerTestCoverageTest to check whether tests based on
> SerializerTestBase and TypeSerializerUpgradeTestBase
> ---
>
> Key: FLINK-19984
> URL: https://issues.apache.org/jira/browse/FLINK-19984
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.12.0
>Reporter: Nicholas Jiang
>Assignee: Nicholas Jiang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently, we have {{TypeInfoTestCoverageTest}} which checks that we have a
> test that extends {{TypeInformationTestBase}} for all type infos. But
> {{TypeSerializer}} doesn’t have the same thing that would verify that
> {{TypeSerializer}} has tests that extend {{SerializerTestBase}} and
> {{TypeSerializerUpgradeTestBase}}. Therefore we don’t know if test coverage
> of {{TypeSerializer}} is good.
> This would add {{TypeSerializerTestCoverageTest}} to check whether to have
> tests based on {{SerializerTestBase}} and {{TypeSerializerUpgradeTestBase}}
> because all serializers should have tests based on both of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19984) Add TypeSerializerTestCoverageTest to check whether tests based on SerializerTestBase and TypeSerializerUpgradeTestBase
[
https://issues.apache.org/jira/browse/FLINK-19984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-19984.
Resolution: Fixed
master: d9c5b436143d063a2b36950477d2f84e833ddfc6
release-1.12: c40dc919ad76c4e8a3aea58d74e998dee1504e17
> Add TypeSerializerTestCoverageTest to check whether tests based on
> SerializerTestBase and TypeSerializerUpgradeTestBase
> ---
>
> Key: FLINK-19984
> URL: https://issues.apache.org/jira/browse/FLINK-19984
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Reporter: Nicholas Jiang
>Assignee: Nicholas Jiang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently, we have {{TypeInfoTestCoverageTest}} which checks that we have a
> test that extends {{TypeInformationTestBase}} for all type infos. But
> {{TypeSerializer}} doesn’t have the same thing that would verify that
> {{TypeSerializer}} has tests that extend {{SerializerTestBase}} and
> {{TypeSerializerUpgradeTestBase}}. Therefore we don’t know if test coverage
> of {{TypeSerializer}} is good.
> This would add {{TypeSerializerTestCoverageTest}} to check whether to have
> tests based on {{SerializerTestBase}} and {{TypeSerializerUpgradeTestBase}}
> because all serializers should have tests based on both of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20443) ContinuousProcessingTimeTrigger doesn't fire at the end of the window
[
https://issues.apache.org/jira/browse/FLINK-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242385#comment-17242385
]
Aljoscha Krettek commented on FLINK-20443:
--
This is the same as FLINK-5149 so I think it's good to fix it to make them
uniform again.
> ContinuousProcessingTimeTrigger doesn't fire at the end of the window
> -
>
> Key: FLINK-20443
> URL: https://issues.apache.org/jira/browse/FLINK-20443
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Gee
>Priority: Minor
> Labels: pull-request-available
>
>
> {code:java}
> srcStream
> .timeWindowAll(Time.seconds(60))
> .trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(10)))...
> {code}
>
> This can correctly calculate the following interval result : 0-10s 10-20s
> 20-30s 30-40s 40-50s
> But this lost data which was send in 50-60s.
> Because when the first window ends, the time is 59.s, it is not equal to
> window-end-time(60s).So it will not enter the if judgment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20443) ContinuousProcessingTimeTrigger doesn't fire at the end of the window
[
https://issues.apache.org/jira/browse/FLINK-20443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-20443:
-
Summary: ContinuousProcessingTimeTrigger doesn't fire at the end of the
window (was: ContinuousProcessingTimeTrigger lost data in last interval in per
window)
> ContinuousProcessingTimeTrigger doesn't fire at the end of the window
> -
>
> Key: FLINK-20443
> URL: https://issues.apache.org/jira/browse/FLINK-20443
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Gee
>Priority: Minor
> Labels: pull-request-available
>
>
> {code:java}
> srcStream
> .timeWindowAll(Time.seconds(60))
> .trigger(ContinuousProcessingTimeTrigger.of(Time.seconds(10)))...
> {code}
>
> This can correctly calculate the following interval result : 0-10s 10-20s
> 20-30s 30-40s 40-50s
> But this lost data which was send in 50-60s.
> Because when the first window ends, the time is 59.s, it is not equal to
> window-end-time(60s).So it will not enter the if judgment.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20438) SystemProcessingTimeServiceTest fails due to missing output
[
https://issues.apache.org/jira/browse/FLINK-20438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-20438:
-
Summary: SystemProcessingTimeServiceTest fails due to missing output (was:
org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest fails
due to missing output)
> SystemProcessingTimeServiceTest fails due to missing output
> ---
>
> Key: FLINK-20438
> URL: https://issues.apache.org/jira/browse/FLINK-20438
> Project: Flink
> Issue Type: Test
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Matthias
>Priority: Major
> Labels: testability
>
> [This
> build|https://dev.azure.com/mapohl/flink/_build/results?buildId=122=logs=9dc1b5dc-bcfa-5f83-eaa7-0cb181ddc267=ab910030-93db-52a7-74a3-34a0addb481b]
> failed due to
> {{org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest}}
> not getting back, i.e. not producing output, for 900 seconds.
> {noformat}
> # ...
> ==
> Process produced no output for 900 seconds.
> ==
> # ...{noformat}
> The Thread Stacktraces are printed in the build's output. The last test that
> didn't finish is {{SystemProcessingTimeServiceTest}} according to the build
> artifact's {{mvn-2.log}}:
> {noformat}
> # ...
>
> Test
> testShutdownAndWaitPending(org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest)
> is running.
>
> 15:27:56,727 [main] INFO
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest [] -
>
> Test
> testShutdownAndWaitPending(org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest)
> successfully run.
>
> 15:27:56,728 [main] INFO
> org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest [] -
>
> Test
> testQuiesceAndAwaitingCancelsScheduledAtFixRateFuture(org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeServiceTest)
> is running.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20337) Allow stateful migration from StreamingFileSink to FileSink
[ https://issues.apache.org/jira/browse/FLINK-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-20337. Resolution: Implemented release-1.12: d0757e95c3c33d4f13377f2c14c179076b3795b2 (and commits around it) > Allow stateful migration from StreamingFileSink to FileSink > > > Key: FLINK-20337 > URL: https://issues.apache.org/jira/browse/FLINK-20337 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Critical > Fix For: 1.12.0 > > > Flink-1.12 introduces the `FileSink` in FLINK-19510, which can guarantee the > exactly once semantics both in the streaming and batch execution mode. We > need to provide a way for the user who uses `StreamingFileSink` to migrate > from `StreamingFileSink` to `FileSink`. > For this purpose we propose to let the new sink writer operator could load > the previous StreamingFileSink's state and then the `SinkWriter` could have > the opertunity to handle the old state. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20337) Allow stateful migration from StreamingFileSink to FileSink
[ https://issues.apache.org/jira/browse/FLINK-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240799#comment-17240799 ] Aljoscha Krettek commented on FLINK-20337: -- master: da1ad03d68e6c57a95d7375259247053aa8fc24e and commits around it. > Allow stateful migration from StreamingFileSink to FileSink > > > Key: FLINK-20337 > URL: https://issues.apache.org/jira/browse/FLINK-20337 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Critical > Fix For: 1.12.0 > > > Flink-1.12 introduces the `FileSink` in FLINK-19510, which can guarantee the > exactly once semantics both in the streaming and batch execution mode. We > need to provide a way for the user who uses `StreamingFileSink` to migrate > from `StreamingFileSink` to `FileSink`. > For this purpose we propose to let the new sink writer operator could load > the previous StreamingFileSink's state and then the `SinkWriter` could have > the opertunity to handle the old state. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20337) Allow stateful migration from StreamingFileSink to FileSink
[ https://issues.apache.org/jira/browse/FLINK-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20337: - Summary: Allow stateful migration from StreamingFileSink to FileSink (was: Make migrate `StreamingFileSink` to `FileSink` possible) > Allow stateful migration from StreamingFileSink to FileSink > > > Key: FLINK-20337 > URL: https://issues.apache.org/jira/browse/FLINK-20337 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Critical > Fix For: 1.12.0 > > > Flink-1.12 introduces the `FileSink` in FLINK-19510, which can guarantee the > exactly once semantics both in the streaming and batch execution mode. We > need to provide a way for the user who uses `StreamingFileSink` to migrate > from `StreamingFileSink` to `FileSink`. > For this purpose we propose to let the new sink writer operator could load > the previous StreamingFileSink's state and then the `SinkWriter` could have > the opertunity to handle the old state. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Deleted] (FLINK-20338) Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state.
[ https://issues.apache.org/jira/browse/FLINK-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek deleted FLINK-20338: - > Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state. > --- > > Key: FLINK-20338 > URL: https://issues.apache.org/jira/browse/FLINK-20338 > Project: Flink > Issue Type: Sub-task >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Critical > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Deleted] (FLINK-20339) `FileWriter` support to load StreamingFileSink's state.
[ https://issues.apache.org/jira/browse/FLINK-20339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek deleted FLINK-20339: - > `FileWriter` support to load StreamingFileSink's state. > --- > > Key: FLINK-20339 > URL: https://issues.apache.org/jira/browse/FLINK-20339 > Project: Flink > Issue Type: Sub-task >Reporter: Guowei Ma >Assignee: Yun Gao >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20339) `FileWriter` support to load StreamingFileSink's state.
[ https://issues.apache.org/jira/browse/FLINK-20339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239700#comment-17239700 ] Aljoscha Krettek commented on FLINK-20339: -- Just for anyone wondering, this was closed because we now just do the issue in one go and use the parent for tracking. > `FileWriter` support to load StreamingFileSink's state. > --- > > Key: FLINK-20339 > URL: https://issues.apache.org/jira/browse/FLINK-20339 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Yun Gao >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20338) Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state.
[ https://issues.apache.org/jira/browse/FLINK-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20338: - Fix Version/s: (was: 1.12.0) > Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state. > --- > > Key: FLINK-20338 > URL: https://issues.apache.org/jira/browse/FLINK-20338 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Critical > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20338) Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state.
[ https://issues.apache.org/jira/browse/FLINK-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239699#comment-17239699 ] Aljoscha Krettek commented on FLINK-20338: -- Just for anyone wondering, this was closed because we now just do the issue in one go and use the parent for tracking. > Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state. > --- > > Key: FLINK-20338 > URL: https://issues.apache.org/jira/browse/FLINK-20338 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Critical > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20339) `FileWriter` support to load StreamingFileSink's state.
[ https://issues.apache.org/jira/browse/FLINK-20339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20339: - Fix Version/s: (was: 1.12.0) > `FileWriter` support to load StreamingFileSink's state. > --- > > Key: FLINK-20339 > URL: https://issues.apache.org/jira/browse/FLINK-20339 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Yun Gao >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20339) `FileWriter` support to load StreamingFileSink's state.
[ https://issues.apache.org/jira/browse/FLINK-20339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-20339: Assignee: Yun Gao > `FileWriter` support to load StreamingFileSink's state. > --- > > Key: FLINK-20339 > URL: https://issues.apache.org/jira/browse/FLINK-20339 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Yun Gao >Priority: Critical > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20338) Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state.
[ https://issues.apache.org/jira/browse/FLINK-20338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-20338: Assignee: Guowei Ma > Make the `StatefulSinkWriterOperator` load `StreamingFileSink`'s state. > --- > > Key: FLINK-20338 > URL: https://issues.apache.org/jira/browse/FLINK-20338 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20360) AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks is unstable
[
https://issues.apache.org/jira/browse/FLINK-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-20360.
Fix Version/s: (was: 1.12.0)
Resolution: Not A Problem
The change is not unrelated, some code relies on the fact that the serialized
value is not cached.
This code is not on {{master}}. right? So it's ok to close this issue.
> AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks
> is unstable
>
>
> Key: FLINK-20360
> URL: https://issues.apache.org/jira/browse/FLINK-20360
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: test-stability
>
> This is a CI PR run, but the change is unrelated:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10119=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.046
> s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest
> [ERROR]
> testPeriodicWatermarks[0](org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite)
> Time elapsed: 0.004 s <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<102>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks(AbstractFetcherWatermarksTest.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-20360) AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks is unstable
[
https://issues.apache.org/jira/browse/FLINK-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239665#comment-17239665
]
Aljoscha Krettek edited comment on FLINK-20360 at 11/27/20, 12:53 PM:
--
The change is not unrelated, some code relies on the fact that the serialized
value is not cached.
This code is not on {{master}}, right? So it's ok to close this issue.
was (Author: aljoscha):
The change is not unrelated, some code relies on the fact that the serialized
value is not cached.
This code is not on {{master}}. right? So it's ok to close this issue.
> AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks
> is unstable
>
>
> Key: FLINK-20360
> URL: https://issues.apache.org/jira/browse/FLINK-20360
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: test-stability
>
> This is a CI PR run, but the change is unrelated:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10119=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.046
> s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest
> [ERROR]
> testPeriodicWatermarks[0](org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite)
> Time elapsed: 0.004 s <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<102>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks(AbstractFetcherWatermarksTest.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20360) AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks is unstable
[
https://issues.apache.org/jira/browse/FLINK-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239656#comment-17239656
]
Aljoscha Krettek commented on FLINK-20360:
--
I'm taking a look.
> AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks
> is unstable
>
>
> Key: FLINK-20360
> URL: https://issues.apache.org/jira/browse/FLINK-20360
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> This is a CI PR run, but the change is unrelated:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10119=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.046
> s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest
> [ERROR]
> testPeriodicWatermarks[0](org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite)
> Time elapsed: 0.004 s <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<102>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks(AbstractFetcherWatermarksTest.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20360) AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks is unstable
[
https://issues.apache.org/jira/browse/FLINK-20360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reassigned FLINK-20360:
Assignee: Aljoscha Krettek
> AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks
> is unstable
>
>
> Key: FLINK-20360
> URL: https://issues.apache.org/jira/browse/FLINK-20360
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> This is a CI PR run, but the change is unrelated:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10119=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> [ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.046
> s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest
> [ERROR]
> testPeriodicWatermarks[0](org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite)
> Time elapsed: 0.004 s <<< FAILURE!
> java.lang.AssertionError: expected:<3> but was:<102>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcherWatermarksTest$PeriodicWatermarksSuite.testPeriodicWatermarks(AbstractFetcherWatermarksTest.java:139)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20190) A New Window Trigger that can trigger window operation both by event time interval、event count for DataStream API
[
https://issues.apache.org/jira/browse/FLINK-20190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239655#comment-17239655
]
Aljoscha Krettek commented on FLINK-20190:
--
I can probably comment once I understand your use case better. Why are you
using {{ContinuousEventTimeTrigger}}`? It would help to outline what the
expected input is and then also what the expected output is, maybe with some
example watermarks and what the expected output would be for them.
> A New Window Trigger that can trigger window operation both by event time
> interval、event count for DataStream API
> -
>
> Key: FLINK-20190
> URL: https://issues.apache.org/jira/browse/FLINK-20190
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream
>Reporter: GaryGao
>Priority: Major
> Fix For: 1.12.0
>
>
> In production environment, when we are do some window operation, such as
> window aggregation, using data stream api, developers are always asked to not
> only trigger the window operation when the watermark pass the max timestamp
> of window, but also trigger it both by fixed event time interval and fixed
> count of event.The reason why we want to do this is we are looking forward to
> get the frequently updated window operation result, other than waiting for a
> long time until the watermark pass the max timestamp of window.This is very
> useful in reporting and other BI applications.
> For now the default triggers provided by flink can not close this
> requirement, so I developed a New Trigger, so called
> CountAndContinuousEventTimeTrigger, combine ContinuousEventTimeTrigger with
> CountTrigger to do the above thing.
>
> To use CountAndContinuousEventTimeTrigger, you should specify two parameters
> as revealed in it constructor:
> {code:java}
> private CountAndContinuousEventTimeTrigger(Time interval, long
> maxCount);{code}
> * Time interval, it means this trigger will continuously fires based on a
> given time interval, the same as ContinuousEventTimeTrigger.
> * long maxCount, it means this trigger will fires once the count of elements
> in a pane reaches the given count, the same as CountTrigger.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20302) Recommend DataStream API with BATCH execution mode in DataSet docs
[ https://issues.apache.org/jira/browse/FLINK-20302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-20302. Fix Version/s: 1.12.0 Resolution: Done master: 01d8f9967e27c1a1c154cb3fdee9ced2fd315321 > Recommend DataStream API with BATCH execution mode in DataSet docs > -- > > Key: FLINK-20302 > URL: https://issues.apache.org/jira/browse/FLINK-20302 > Project: Flink > Issue Type: Sub-task > Components: API / DataSet, API / DataStream, Documentation >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20302) Recommend DataStream API with BATCH execution mode in DataSet docs
[ https://issues.apache.org/jira/browse/FLINK-20302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20302: - Summary: Recommend DataStream API with BATCH execution mode in DataSet docs (was: Suggest DataStream API with BATCH execution mode in DataSet docs) > Recommend DataStream API with BATCH execution mode in DataSet docs > -- > > Key: FLINK-20302 > URL: https://issues.apache.org/jira/browse/FLINK-20302 > Project: Flink > Issue Type: Sub-task > Components: API / DataSet, API / DataStream, Documentation >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-19316) FLIP-134: Batch execution for the DataStream API
[ https://issues.apache.org/jira/browse/FLINK-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-19316. Resolution: Done All subtasks have been resolved. > FLIP-134: Batch execution for the DataStream API > > > Key: FLINK-19316 > URL: https://issues.apache.org/jira/browse/FLINK-19316 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Aljoscha Krettek >Assignee: Kostas Kloudas >Priority: Major > > Umbrella issue for [FLIP-134|https://cwiki.apache.org/confluence/x/4i94CQ] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20153) Add documentation for BATCH execution mode
[ https://issues.apache.org/jira/browse/FLINK-20153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-20153. Resolution: Done master: d222ce46b33ab686c503906bf4789c78107608c7 > Add documentation for BATCH execution mode > -- > > Key: FLINK-20153 > URL: https://issues.apache.org/jira/browse/FLINK-20153 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20343) Add overview / reference architecture documentation page
[ https://issues.apache.org/jira/browse/FLINK-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20343: - Summary: Add overview / reference architecture documentation page (was: Add overview / reference architecture page) > Add overview / reference architecture documentation page > > > Key: FLINK-20343 > URL: https://issues.apache.org/jira/browse/FLINK-20343 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Runtime / Coordination >Reporter: Robert Metzger >Assignee: Robert Metzger >Priority: Major > Fix For: 1.12.0 > > > To properly guide users, we should add some generic overview of the > deployment concepts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20342) Revisit page structure in documentation
[ https://issues.apache.org/jira/browse/FLINK-20342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-20342: - Summary: Revisit page structure in documentation (was: Revisit page structure) > Revisit page structure in documentation > --- > > Key: FLINK-20342 > URL: https://issues.apache.org/jira/browse/FLINK-20342 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Runtime / Coordination >Reporter: Robert Metzger >Assignee: Till Rohrmann >Priority: Major > Fix For: 1.12.0 > > > Clean up page structure -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20337) Make migrate `StreamingFileSink` to `FileSink` possible
[
https://issues.apache.org/jira/browse/FLINK-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238729#comment-17238729
]
Aljoscha Krettek commented on FLINK-20337:
--
It's not incomplete, it's just that existing users of the {{StreamingFileSink}}
cannot migrate. This doesn't have to be a blocker. We can add it whenever it's
ready and ship it in a point release.
> Make migrate `StreamingFileSink` to `FileSink` possible
> ---
>
> Key: FLINK-20337
> URL: https://issues.apache.org/jira/browse/FLINK-20337
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Guowei Ma
>Assignee: Guowei Ma
>Priority: Blocker
> Fix For: 1.12.0
>
>
> Flink-1.12 introduces the `FileSink` in FLINK-19510, which can guarantee the
> exactly once semantics both in the streaming and batch execution mode. We
> need to provide a way for the user who uses `StreamingFileSink` to migrate
> from `StreamingFileSink` to `FileSink`.
> For this purpose we propose to let the new sink writer operator could load
> the previous StreamingFileSink's state and then the `SinkWriter` could have
> the opertunity to handle the old state.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20337) Make migrate `StreamingFileSink` to `FileSink` possible
[ https://issues.apache.org/jira/browse/FLINK-20337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek reassigned FLINK-20337: Assignee: Guowei Ma > Make migrate `StreamingFileSink` to `FileSink` possible > --- > > Key: FLINK-20337 > URL: https://issues.apache.org/jira/browse/FLINK-20337 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Blocker > Fix For: 1.12.0 > > > Flink-1.12 introduces the `FileSink` in FLINK-19510, which can guarantee the > exactly once semantics both in the streaming and batch execution mode. We > need to provide a way for the user who uses `StreamingFileSink` to migrate > from `StreamingFileSink` to `FileSink`. > For this purpose we propose to let the new sink writer operator could load > the previous StreamingFileSink's state and then the `SinkWriter` could have > the opertunity to handle the old state. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20329) Elasticsearch7DynamicSinkITCase hangs
[
https://issues.apache.org/jira/browse/FLINK-20329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238628#comment-17238628
]
Aljoscha Krettek commented on FLINK-20329:
--
Probably related to FLINK-20279. It seems using testcontainers still has some
initial problems.
> Elasticsearch7DynamicSinkITCase hangs
> -
>
> Key: FLINK-20329
> URL: https://issues.apache.org/jira/browse/FLINK-20329
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10052=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20
> {code}
> 2020-11-24T16:04:05.9260517Z [INFO] Running
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch7DynamicSinkITCase
> 2020-11-24T16:19:25.5481231Z
> ==
> 2020-11-24T16:19:25.5483549Z Process produced no output for 900 seconds.
> 2020-11-24T16:19:25.5484064Z
> ==
> 2020-11-24T16:19:25.5484498Z
> ==
> 2020-11-24T16:19:25.5484882Z The following Java processes are running (JPS)
> 2020-11-24T16:19:25.5485475Z
> ==
> 2020-11-24T16:19:25.5694497Z Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> 2020-11-24T16:19:25.7263048Z 16192 surefirebooter5057948964630155904.jar
> 2020-11-24T16:19:25.7263515Z 18566 Jps
> 2020-11-24T16:19:25.7263709Z 959 Launcher
> 2020-11-24T16:19:25.7411148Z
> ==
> 2020-11-24T16:19:25.7427013Z Printing stack trace of Java process 16192
> 2020-11-24T16:19:25.7427369Z
> ==
> 2020-11-24T16:19:25.7484365Z Picked up JAVA_TOOL_OPTIONS:
> -XX:+HeapDumpOnOutOfMemoryError
> 2020-11-24T16:19:26.0848776Z 2020-11-24 16:19:26
> 2020-11-24T16:19:26.0849578Z Full thread dump OpenJDK 64-Bit Server VM
> (25.275-b01 mixed mode):
> 2020-11-24T16:19:26.0849831Z
> 2020-11-24T16:19:26.0850185Z "Attach Listener" #32 daemon prio=9 os_prio=0
> tid=0x7fc148001000 nid=0x48e7 waiting on condition [0x]
> 2020-11-24T16:19:26.0850595Zjava.lang.Thread.State: RUNNABLE
> 2020-11-24T16:19:26.0850814Z
> 2020-11-24T16:19:26.0851375Z "testcontainers-ryuk" #31 daemon prio=5
> os_prio=0 tid=0x7fc251232000 nid=0x3fb0 in Object.wait()
> [0x7fc1012c4000]
> 2020-11-24T16:19:26.0854688Zjava.lang.Thread.State: TIMED_WAITING (on
> object monitor)
> 2020-11-24T16:19:26.0855379Z at java.lang.Object.wait(Native Method)
> 2020-11-24T16:19:26.0855844Z at
> org.testcontainers.utility.ResourceReaper.lambda$null$1(ResourceReaper.java:142)
> 2020-11-24T16:19:26.0857272Z - locked <0x8e2bd2d0> (a
> java.util.ArrayList)
> 2020-11-24T16:19:26.0857977Z at
> org.testcontainers.utility.ResourceReaper$$Lambda$93/1981729428.run(Unknown
> Source)
> 2020-11-24T16:19:26.0858471Z at
> org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady(RateLimiter.java:27)
> 2020-11-24T16:19:26.0858961Z at
> org.testcontainers.utility.ResourceReaper.lambda$start$2(ResourceReaper.java:133)
> 2020-11-24T16:19:26.0859422Z at
> org.testcontainers.utility.ResourceReaper$$Lambda$92/40191541.run(Unknown
> Source)
> 2020-11-24T16:19:26.0859788Z at java.lang.Thread.run(Thread.java:748)
> 2020-11-24T16:19:26.0860030Z
> 2020-11-24T16:19:26.0860371Z "process reaper" #24 daemon prio=10 os_prio=0
> tid=0x7fc0f803b800 nid=0x3f92 waiting on condition [0x7fc10296e000]
> 2020-11-24T16:19:26.0860913Zjava.lang.Thread.State: TIMED_WAITING
> (parking)
> 2020-11-24T16:19:26.0861387Z at sun.misc.Unsafe.park(Native Method)
> 2020-11-24T16:19:26.0862495Z - parking to wait for <0x8814bf30> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
> 2020-11-24T16:19:26.0863253Z at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> 2020-11-24T16:19:26.0863760Z at
> java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> 2020-11-24T16:19:26.0864274Z at
> java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> 2020-11-24T16:19:26.0864762Z at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> 2020-11-24T16:19:26.0865299Z at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> 2020-11-24T16:19:26.0866000Z at
[jira] [Closed] (FLINK-19510) FLIP-143: Introduce a new unified sink api
[ https://issues.apache.org/jira/browse/FLINK-19510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek closed FLINK-19510. Resolution: Implemented All subtasks have been implemented > FLIP-143: Introduce a new unified sink api > -- > > Key: FLINK-19510 > URL: https://issues.apache.org/jira/browse/FLINK-19510 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > As described in FLIP-143: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API > we want to introduce a new unified sink api. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20304) Fail hard when trying to run job with broadcast state in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reassigned FLINK-20304:
Assignee: Kostas Kloudas
> Fail hard when trying to run job with broadcast state in BATCH execution mode
> -
>
> Key: FLINK-20304
> URL: https://issues.apache.org/jira/browse/FLINK-20304
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Assignee: Kostas Kloudas
>Priority: Critical
> Fix For: 1.12.0
>
>
> Contrary to the documentation it is currently possible to run a job with
> broadcast state in {{BATCH}} execution mode. Since accessing the keyed state
> from the broadcast side fails, we shouldn't allow the submissions of these
> kind of jobs in the first place. Hence, I would suggest to fail hard if one
> tries to run a job using the broadcast state pattern in {{BATCH}} execution
> mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20304) Fail hard when trying to run job with broadcast state in BATCH execution mode
[
https://issues.apache.org/jira/browse/FLINK-20304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238018#comment-17238018
]
Aljoscha Krettek commented on FLINK-20304:
--
I think it's not a blocker but still nice to have.
> Fail hard when trying to run job with broadcast state in BATCH execution mode
> -
>
> Key: FLINK-20304
> URL: https://issues.apache.org/jira/browse/FLINK-20304
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.0
>Reporter: Till Rohrmann
>Priority: Critical
> Fix For: 1.12.0
>
>
> Contrary to the documentation it is currently possible to run a job with
> broadcast state in {{BATCH}} execution mode. Since accessing the keyed state
> from the broadcast side fails, we shouldn't allow the submissions of these
> kind of jobs in the first place. Hence, I would suggest to fail hard if one
> tries to run a job using the broadcast state pattern in {{BATCH}} execution
> mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19697) Make the Sink API committer retry-able
[ https://issues.apache.org/jira/browse/FLINK-19697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-19697: - Parent: (was: FLINK-19510) Issue Type: New Feature (was: Sub-task) > Make the Sink API committer retry-able > -- > > Key: FLINK-19697 > URL: https://issues.apache.org/jira/browse/FLINK-19697 > Project: Flink > Issue Type: New Feature > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Guowei Ma >Assignee: Guowei Ma >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17691) FlinkKafkaProducer transactional.id too long when using Semantic.EXACTLY_ONCE
[
https://issues.apache.org/jira/browse/FLINK-17691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-17691.
Fix Version/s: 1.12.0
Resolution: Fixed
master: 8e402832caae9ea40d665cecb7b09204e1530e03
> FlinkKafkaProducer transactional.id too long when using Semantic.EXACTLY_ONCE
> -
>
> Key: FLINK-17691
> URL: https://issues.apache.org/jira/browse/FLINK-17691
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.10.0, 1.11.0
>Reporter: freezhan
>Assignee: John Mathews
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments: image-2020-05-14-20-43-57-414.png,
> image-2020-05-14-20-45-24-030.png, image-2020-05-14-20-45-59-878.png,
> image-2020-05-14-21-09-01-906.png, image-2020-05-14-21-16-43-810.png,
> image-2020-05-14-21-17-09-784.png
>
>
> When sink to Kafka using the {color:#FF}Semantic.EXACTLY_ONCE {color}mode.
> The flink Kafka Connector Producer will auto set the
> {color:#FF}transactional.id{color}, and the user - defined value are
> ignored.
>
> When the job operator name too long, will send failed
> transactional.id is exceeds the kafka {color:#FF}coordinator_key{color}
> limit
> !image-2020-05-14-21-09-01-906.png!
>
> *The flink Kafka Connector policy for automatic generation of transaction.id
> is as follows*
>
> 1. use the {color:#FF}taskName + "-" + operatorUniqueID{color} as
> transactional.id prefix (may be too long)
> getRuntimeContext().getTaskName() + "-" + ((StreamingRuntimeContext)
> getRuntimeContext()).getOperatorUniqueID()
> 2. Range of available transactional ids
> [nextFreeTransactionalId, nextFreeTransactionalId + parallelism *
> kafkaProducersPoolSize)
> !image-2020-05-14-20-43-57-414.png!
> !image-2020-05-14-20-45-24-030.png!
> !image-2020-05-14-20-45-59-878.png!
>
> *The Kafka transaction.id check policy as follows:*
>
> {color:#FF}string bytes.length can't larger than Short.MAX_VALUE
> (32767){color}
> !image-2020-05-14-21-16-43-810.png!
> !image-2020-05-14-21-17-09-784.png!
>
> *To reproduce this bug, the following conditions must be met:*
>
> # send msg to kafka with exactly once mode
> # the task TaskName' length + TaskName's length is lager than the 32767 (A
> very long line of SQL or window statements can appear)
> *I suggest a solution:*
>
> 1. Allows users to customize transactional.id 's prefix
> or
> 2. Do md5 on the prefix before returning the real transactional.id
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
