[GitHub] [flink] flinkbot commented on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
flinkbot commented on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526487220 ## CI report: * 3c6c3d69983136fb8bfa11b28a7ba783dfe61e52 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526487220 ## CI report: * 3c6c3d69983136fb8bfa11b28a7ba783dfe61e52 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/125188905) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one
flinkbot edited a comment on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one URL: https://github.com/apache/flink/pull/9252#issuecomment-515824110 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b817533cb971312f24da0fd9c64fb04597b70d79 (Fri Aug 30 07:39:14 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-13514) StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
[
https://issues.apache.org/jira/browse/FLINK-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek closed FLINK-13514.
Resolution: Fixed
> StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
> -
>
> Key: FLINK-13514
> URL: https://issues.apache.org/jira/browse/FLINK-13514
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task, Tests
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Chesnay Schepler
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.0, 1.9.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Somewhat reproducible by running the test locally a few hundred times.
> {code:java}
> org.mockito.exceptions.verification.NeverWantedButInvoked:
> keyedStateHandle.discardState();
> Never wanted here:
> -> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge(StreamTaskTest.java:543)
> But invoked here:
> -> at
> org.apache.flink.runtime.state.SnapshotResult.discardState(SnapshotResult.java:82)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge(StreamTaskTest.java:543)
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9336: [FLINK-13548][Deployment/YARN]Support priority of the Flink YARN application
flinkbot edited a comment on issue #9336: [FLINK-13548][Deployment/YARN]Support priority of the Flink YARN application URL: https://github.com/apache/flink/pull/9336#issuecomment-517608148 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3981f27cdd12ae38fa03041779ed159891c102fa (Fri Aug 30 08:06:44 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13514) StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
[
https://issues.apache.org/jira/browse/FLINK-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919304#comment-16919304
]
Aljoscha Krettek commented on FLINK-13514:
--
Fixed on release-1.9 in 3d71518ef9c96bc8fe0add3b4c25bf141aa599db
> StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
> -
>
> Key: FLINK-13514
> URL: https://issues.apache.org/jira/browse/FLINK-13514
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task, Tests
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Chesnay Schepler
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.0, 1.9.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Somewhat reproducible by running the test locally a few hundred times.
> {code:java}
> org.mockito.exceptions.verification.NeverWantedButInvoked:
> keyedStateHandle.discardState();
> Never wanted here:
> -> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge(StreamTaskTest.java:543)
> But invoked here:
> -> at
> org.apache.flink.runtime.state.SnapshotResult.discardState(SnapshotResult.java:82)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge(StreamTaskTest.java:543)
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-13514) StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
[
https://issues.apache.org/jira/browse/FLINK-13514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919301#comment-16919301
]
Aljoscha Krettek commented on FLINK-13514:
--
Fixed on master in 60b65c42ca2709d20cb59f1617d96a80ec870b6c
> StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge unstable
> -
>
> Key: FLINK-13514
> URL: https://issues.apache.org/jira/browse/FLINK-13514
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task, Tests
>Affects Versions: 1.9.0, 1.10.0
>Reporter: Chesnay Schepler
>Assignee: Aljoscha Krettek
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.0, 1.9.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Somewhat reproducible by running the test locally a few hundred times.
> {code:java}
> org.mockito.exceptions.verification.NeverWantedButInvoked:
> keyedStateHandle.discardState();
> Never wanted here:
> -> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge(StreamTaskTest.java:543)
> But invoked here:
> -> at
> org.apache.flink.runtime.state.SnapshotResult.discardState(SnapshotResult.java:82)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge(StreamTaskTest.java:543)
> at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Reopened] (FLINK-13791) Speed up sidenav by using group_by
[
https://issues.apache.org/jira/browse/FLINK-13791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reopened FLINK-13791:
--
Reverted due to FLINK-13907.
> Speed up sidenav by using group_by
> --
>
> Key: FLINK-13791
> URL: https://issues.apache.org/jira/browse/FLINK-13791
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {{_includes/sidenav.html}} parses through {{pages_by_language}} over and over
> again trying to find children when building the (recursive) side navigation.
> We could do this once with a {{group_by}} instead.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Reopened] (FLINK-13726) Build docs with jekyll 4.0.0.pre.beta1
[ https://issues.apache.org/jira/browse/FLINK-13726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reopened FLINK-13726: -- Reverted due to FLINK-13907. > Build docs with jekyll 4.0.0.pre.beta1 > -- > > Key: FLINK-13726 > URL: https://issues.apache.org/jira/browse/FLINK-13726 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.10.0 >Reporter: Nico Kruber >Assignee: Nico Kruber >Priority: Major > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Jekyll 4 is way faster in generating the docs than jekyll 3 - probably due to > the newly introduced cache. Site generation time goes down by roughly a > factor of 2.5 even with the current beta version! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Reopened] (FLINK-13725) Use sassc for faster doc generation
[
https://issues.apache.org/jira/browse/FLINK-13725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler reopened FLINK-13725:
--
Reverted due to FLINK-13907.
> Use sassc for faster doc generation
> ---
>
> Key: FLINK-13725
> URL: https://issues.apache.org/jira/browse/FLINK-13725
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Affects Versions: 1.10.0
>Reporter: Nico Kruber
>Assignee: Nico Kruber
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Jekyll requires {{sass}} but can optionally also use a C-based implementation
> provided by {{sassc}}. Although we do not use sass directly, there may be
> some indirect use inside jekyll. It doesn't seem to hurt to upgrade here.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (FLINK-13907) Master documentation cannot be built
[ https://issues.apache.org/jira/browse/FLINK-13907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-13907. Resolution: Fixed > Master documentation cannot be built > > > Key: FLINK-13907 > URL: https://issues.apache.org/jira/browse/FLINK-13907 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.10.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Fix For: 1.10.0 > > > The documentation currently cannot be built on buildbot because due to recent > changes we require a newer ruby version. > While a newer ruby version is installed, I can't find a way to actually > activate it. > For the time being I'll revert these changes: > * ef74a61f54f190926a8388f46db7919e0e94420b > * 065de4b573a05b0c3436ff2d3af3e0c16589a1a7 > * f802e16b06b0c3a3682af7f9017f9c0a69e5d4de > * ac1b8dbf15c405d0646671a138a53c9953153165 > * c64e167b8003b7379545c1b83e54d9491164b7a8 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (FLINK-12501) AvroTypeSerializer does not work with types generated by avrohugger
[
https://issues.apache.org/jira/browse/FLINK-12501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reassigned FLINK-12501:
Assignee: Aljoscha Krettek
> AvroTypeSerializer does not work with types generated by avrohugger
> ---
>
> Key: FLINK-12501
> URL: https://issues.apache.org/jira/browse/FLINK-12501
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Reporter: Aljoscha Krettek
>Assignee: Aljoscha Krettek
>Priority: Major
>
> The main problem is that the code in {{SpecificData.createSchema()}} tries to
> reflectively read the {{SCHEMA$}} field, that is normally there in Avro
> generated classes. However, avrohugger generates this field in a companion
> object, which the reflective Java code will therefore not find.
> This is also described in these ML threads:
> *
> [https://lists.apache.org/thread.html/5db58c7d15e4e9aaa515f935be3b342fe036e97d32e1fb0f0d1797ee@%3Cuser.flink.apache.org%3E]
> *
> [https://lists.apache.org/thread.html/cf1c5b8fa7f095739438807de9f2497e04ffe55237c5dea83355112d@%3Cuser.flink.apache.org%3E]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-13910) Many serializable classes have no explicit 'serialVersionUID'
[
https://issues.apache.org/jira/browse/FLINK-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang updated FLINK-13910:
-
Fix Version/s: 1.9.1
1.10.0
> Many serializable classes have no explicit 'serialVersionUID'
> -
>
> Key: FLINK-13910
> URL: https://issues.apache.org/jira/browse/FLINK-13910
> Project: Flink
> Issue Type: Bug
>Reporter: Yun Tang
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
> Attachments: SerializableNoSerialVersionUIDField
>
>
> Currently, many serializable classes in Flink have no explicit
> 'serialVersionUID'. As [official
> doc|https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization]
> said, {{Serializable classes must define a Serial Version UID}}.
> No 'serialVersionUID' would cause compatibility problem. Take
> {{TwoPhaseCommitSinkFunction}} for example, since no explicit
> 'serialVersionUID' defined, after
> [FLINK-10455|https://github.com/apache/flink/commit/489be82a6d93057ed4a3f9bf38ef50d01d11d96b]
> introduced, its default 'serialVersionUID' has changed from
> "4584405056408828651" to "4064406918549730832". In other words, if we submit
> a job from Flink-1.6.3 local home to remote Flink-1.6.2 cluster with the
> usage of {{TwoPhaseCommitSinkFunction}}, we would get exception like:
> {code:java}
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
> instantiate user function.
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:104)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException:
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction;
> local class incompatible: stream classdesc serialVersionUID =
> 4584405056408828651, local class serialVersionUID = 4064406918549730832
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:537)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:512)
> at
> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:473)
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:224)
> ... 4 more
> {code}
> Similar problems existed in
> {{org.apache.flink.streaming.api.operators.SimpleOperatorFactory}} which has
> different 'serialVersionUID' from release-1.9 and current master branch.
> IMO, we might have two options to fix this bug:
> # Add explicit serialVersionUID for those classes which is identical to
> latest Flink-1.9.0 release code.
> # Use similar mechanism like {{FailureTolerantObjectInputStream}} in
> {{InstantiationUtil}} to ignore serialVersionUID mismatch.
> I have collected all production classes without serialVersionUID from latest
> master branch in the attachment, which counts to 639 classes.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] zentol commented on issue #9494: [FLINK-12847] [Connectors / Kinesis] update flink-connector-kinesis to use Apache 2.0 licensed code
zentol commented on issue #9494: [FLINK-12847] [Connectors / Kinesis] update flink-connector-kinesis to use Apache 2.0 licensed code URL: https://github.com/apache/flink/pull/9494#issuecomment-526515811 I'll try to make another pass later today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13910) Many serializable classes have no explicit 'serialVersionUID'
[
https://issues.apache.org/jira/browse/FLINK-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919337#comment-16919337
]
Dawid Wysakowicz commented on FLINK-13910:
--
Just my two cents. To start with I completely agree that all Serializable
classes should have a serialVersionUID, and it is a really good idea to somehow
check for it automatically.
Having said that I don't think we should update all 640 classes that violate it
as of now. I think it does not pose any serious danger unless they take part in
checkpointing/savepointing mechanism. In the end we do not guarantee binary
compatibility for classes that are not annotated with {{@Public}} annotation.
We could solve such issues as the {{TwoPhaseCommitSinkFunction}} in a case by
case manner.
> Many serializable classes have no explicit 'serialVersionUID'
> -
>
> Key: FLINK-13910
> URL: https://issues.apache.org/jira/browse/FLINK-13910
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Reporter: Yun Tang
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
> Attachments: SerializableNoSerialVersionUIDField
>
>
> Currently, many serializable classes in Flink have no explicit
> 'serialVersionUID'. As [official
> doc|https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization]
> said, {{Serializable classes must define a Serial Version UID}}.
> No 'serialVersionUID' would cause compatibility problem. Take
> {{TwoPhaseCommitSinkFunction}} for example, since no explicit
> 'serialVersionUID' defined, after
> [FLINK-10455|https://github.com/apache/flink/commit/489be82a6d93057ed4a3f9bf38ef50d01d11d96b]
> introduced, its default 'serialVersionUID' has changed from
> "4584405056408828651" to "4064406918549730832". In other words, if we submit
> a job from Flink-1.6.3 local home to remote Flink-1.6.2 cluster with the
> usage of {{TwoPhaseCommitSinkFunction}}, we would get exception like:
> {code:java}
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
> instantiate user function.
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:104)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException:
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction;
> local class incompatible: stream classdesc serialVersionUID =
> 4584405056408828651, local class serialVersionUID = 4064406918549730832
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:537)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
> at
> org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:512)
> at
> org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:473)
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:224)
> ... 4 more
> {code}
> Similar problems existed in
> {{org.apache.flink.streaming.api.operators.SimpleOperatorFactory}} which has
> different 'serialVersionUID' from release-1.9 and current master branch.
> IMO, we might have two options to fix this bug:
> # Add explicit serialVersionUID for those classes which is identical to
> latest Flink-1.9.0 release code.
> # Use similar mechanism like {{FailureTolerantObjectInputStream}} in
> {{InstantiationUtil}} to
[GitHub] [flink] flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions URL: https://github.com/apache/flink/pull/9562#issuecomment-526200719 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4d5aa4e6bbd588fdaa1cab4b25eb6e1ed3eabf8e (Fri Aug 30 08:54:34 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319421522
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -1368,13 +1369,19 @@ private void
checkpointStreamOperator(StreamOperator op) throws Exception {
private class TimerInvocationContext implements
SystemProcessingTimeService.ScheduledCallbackExecutionContext {
@Override
- public void invoke(ProcessingTimeCallback callback, long
timestamp) {
- synchronized (getCheckpointLock()) {
- try {
- callback.onProcessingTime(timestamp);
- } catch (Throwable t) {
- handleAsyncException("Caught exception
while processing timer.", new TimerException(t));
- }
+ public void invoke(ProcessingTimeCallback callback, long
timestamp) throws InterruptedException {
+ try {
+
mailboxProcessor.getMailboxExecutor(TaskMailbox.MAX_PRIORITY).execute(() -> {
+ synchronized (getCheckpointLock()) {
+ try {
+
callback.onProcessingTime(timestamp);
+ } catch (Throwable t) {
+
handleAsyncException("Caught exception while processing timer.", new
TimerException(t));
+ }
+ }
+ });
+ } catch (Throwable t) {
Review comment:
Do we need the outer exception handling? If the mailbox is incorrect state,
then I'd image that the stack trace is already good enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319426089
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -428,6 +429,10 @@ public final void invoke() throws Exception {
// See FLINK-7430
isRunning = false;
}
+ MailboxExecutor mainMailboxExecutor =
mailboxProcessor.getMainMailboxExecutor();
Review comment:
I'd rather add a method drain to mailboxProcessor that does the following
lines.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#discussion_r319432890 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ## @@ -27,7 +27,6 @@ import org.apache.flink.configuration.TaskManagerOptions; Review comment: Could you add a longer description to the commit message, why it's not needed anymore? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526487220 ## CI report: * 3c6c3d69983136fb8bfa11b28a7ba783dfe61e52 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125188905) * a9d65d8fabd938d47abd801a4686cb185f2fcf68 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-13911) Add the interface of ModelDataConverter, and several base classes that implement this interface.
Xu Yang created FLINK-13911: --- Summary: Add the interface of ModelDataConverter, and several base classes that implement this interface. Key: FLINK-13911 URL: https://issues.apache.org/jira/browse/FLINK-13911 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Reporter: Xu Yang ModelDataConverter is an interface that defines methods for converting a generic model data object to a collection of rows and the other way around. It is used when we are exporting the model data to a table, as well as loading model data from a table for inference. * Add ModelDataConverter which defines the model data conversion methods. * Add SimpleModelDataConverter which is a specific model data converter where the model data is represented as meta data and a collection of strings. * Add LabeledModelDataConverter which is a specific model data converter where the model data is represented as meta data, a collection of strings, and a collection of label values. * Add RichModelDataConverter which is a specific model data converter where the model data is represented as meta data, a collection of strings, and a collection of auxiliary rows. * Add ModelConverterUtils which provides utility functions for implementing the above model data converters. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#discussion_r319420222 ## File path: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/operators/async/ExecutorServiceRule.java ## @@ -0,0 +1,152 @@ +/* Review comment: I think this file is not used in the current version. It would also make sense to just put it into test-utils-junit as a general building block. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319438933
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java
##
@@ -1183,8 +1183,12 @@ public void notifyCheckpointComplete(final long
checkpointID) {
@Override
public void run() {
try {
-
invokable.notifyCheckpointComplete(checkpointID);
- } catch (Throwable t) {
+
invokable.notifyCheckpointCompleteAsync(checkpointID);
Review comment:
Is the future used at all? Do we need a #get here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319416787
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/SystemProcessingTimeServiceTest.java
##
@@ -391,61 +384,6 @@ public void onProcessingTime(long timestamp) throws
Exception {}
}
}
- @Test
Review comment:
Couldn't we keep the two tests?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319416160
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SystemProcessingTimeService.java
##
@@ -139,8 +130,7 @@ else if (status == STATUS_SHUTDOWN) {
// we directly try to register the timer and only react to the
status on exception
// that way we save unnecessary volatile accesses for each timer
try {
- return timerService.scheduleAtFixedRate(
- new RepeatedTriggerTask(status, task,
checkpointLock, callback, nextTimestamp, period),
+ return
timerService.scheduleAtFixedRate(wrapOnTimerCallback(callback, nextTimestamp,
period),
Review comment:
nit: chop
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319431263
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/SynchronousCheckpointITCase.java
##
@@ -151,11 +152,15 @@ protected void performDefaultAction(DefaultActionContext
context) throws Excepti
}
@Override
- public boolean triggerCheckpoint(CheckpointMetaData
checkpointMetaData, CheckpointOptions checkpointOptions, boolean
advanceToEndOfEventTime) throws Exception {
- eventQueue.put(Event.PRE_TRIGGER_CHECKPOINT);
- boolean result =
super.triggerCheckpoint(checkpointMetaData, checkpointOptions,
advanceToEndOfEventTime);
- eventQueue.put(Event.POST_TRIGGER_CHECKPOINT);
- return result;
+ public Future
triggerCheckpointAsync(CheckpointMetaData checkpointMetaData, CheckpointOptions
checkpointOptions, boolean advanceToEndOfEventTime) {
+ try {
+ eventQueue.put(Event.PRE_TRIGGER_CHECKPOINT);
+ Future result =
super.triggerCheckpointAsync(checkpointMetaData, checkpointOptions,
advanceToEndOfEventTime);
+ eventQueue.put(Event.POST_TRIGGER_CHECKPOINT);
+ return result;
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
Review comment:
Swallowing of thread interruption.
Would probably be better to either allow triggerCheckpointAsync to throw
(interrupt)exceptions or at least also call Thread.interrupt().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#discussion_r319425335 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java ## @@ -64,6 +64,7 @@ import org.apache.flink.streaming.runtime.streamstatus.StreamStatusMaintainer; Review comment: Commit message should be a bit more expressive, for example: Drain mailbox while shutting down StreamTask This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319429574
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -841,6 +845,10 @@ protected void declineCheckpoint(long checkpointId) {
new CheckpointException("Task Name" + getName(),
CheckpointFailureReason.CHECKPOINT_DECLINED_TASK_NOT_READY));
}
+ protected void handleCheckpointException(Exception exception) {
+ handleAsyncException(exception);
Review comment:
Why not use existing handleAsyncException and add a message?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319424880
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -428,6 +429,10 @@ public final void invoke() throws Exception {
// See FLINK-7430
isRunning = false;
}
+ MailboxExecutor mainMailboxExecutor =
mailboxProcessor.getMainMailboxExecutor();
+ while (mainMailboxExecutor.tryYield()) {
+ // Run until we have last letters.
Review comment:
Suggestion: // Run until we have processed all pending letters.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#discussion_r319420570 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/nodes/common/CommonLookupJoin.scala ## @@ -17,13 +17,27 @@ */ Review comment: Import order was changed. Can you partially revert that? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319415791
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SystemProcessingTimeService.java
##
@@ -249,89 +239,53 @@ int getNumTasksScheduled() {
//
/**
-* Internal task that is invoked by the timer service and triggers the
target.
+* A context to which {@link ProcessingTimeCallback} would be passed to
be invoked when a timer is up.
*/
- private static final class TriggerTask implements Runnable {
+ public interface ScheduledCallbackExecutionContext {
Review comment:
`@Internal` ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319429689
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -962,6 +970,10 @@ public ProcessingTimeService getProcessingTimeService() {
*/
@Override
public void handleAsyncException(String message, Throwable exception) {
+ handleAsyncException(exception);
+ }
+
+ private void handleAsyncException(Throwable exception) {
if (isRunning) {
// only fail if the task is still running
asyncExceptionHandler.handleAsyncException(message,
exception);
Review comment:
Where is the message coming from?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] AHeise commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
AHeise commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319417621
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SystemProcessingTimeService.java
##
@@ -249,89 +239,53 @@ int getNumTasksScheduled() {
//
/**
-* Internal task that is invoked by the timer service and triggers the
target.
+* A context to which {@link ProcessingTimeCallback} would be passed to
be invoked when a timer is up.
*/
- private static final class TriggerTask implements Runnable {
+ public interface ScheduledCallbackExecutionContext {
- private final AtomicInteger serviceStatus;
- private final Object lock;
- private final ProcessingTimeCallback target;
- private final long timestamp;
- private final AsyncExceptionHandler exceptionHandler;
-
- private TriggerTask(
- final AtomicInteger serviceStatus,
- final AsyncExceptionHandler exceptionHandler,
- final Object lock,
- final ProcessingTimeCallback target,
- final long timestamp) {
-
- this.serviceStatus =
Preconditions.checkNotNull(serviceStatus);
- this.exceptionHandler =
Preconditions.checkNotNull(exceptionHandler);
- this.lock = Preconditions.checkNotNull(lock);
- this.target = Preconditions.checkNotNull(target);
- this.timestamp = timestamp;
- }
+ void invoke(ProcessingTimeCallback callback, long timestamp)
throws InterruptedException;
+ }
- @Override
- public void run() {
- synchronized (lock) {
- try {
- if (serviceStatus.get() ==
STATUS_ALIVE) {
-
target.onProcessingTime(timestamp);
- }
- } catch (Throwable t) {
- TimerException asyncException = new
TimerException(t);
-
exceptionHandler.handleAsyncException("Caught exception while processing
timer.", asyncException);
- }
- }
- }
+ private Runnable wrapOnTimerCallback(ProcessingTimeCallback callback,
long timestamp) {
+ return new TimeTrackingDelegate(status,
callbackExecutionContext, callback, timestamp, 0);
}
- /**
-* Internal task which is repeatedly called by the processing time
service.
-*/
- private static final class RepeatedTriggerTask implements Runnable {
+ private Runnable wrapOnTimerCallback(ProcessingTimeCallback callback,
long nextTimestamp, long period) {
+ return new TimeTrackingDelegate(status,
callbackExecutionContext, callback, nextTimestamp, period);
+ }
+ private static final class TimeTrackingDelegate implements Runnable {
Review comment:
Is this really a delegate?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-3991) Remove deprecated configuration keys from ConfigConstants
[ https://issues.apache.org/jira/browse/FLINK-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-3991: - Component/s: (was: Build System) Runtime / Configuration > Remove deprecated configuration keys from ConfigConstants > - > > Key: FLINK-3991 > URL: https://issues.apache.org/jira/browse/FLINK-3991 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Reporter: Robert Metzger >Priority: Major > Labels: pull-request-available > Fix For: 2.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In > https://github.com/apache/flink/commit/b0acd97935cd21843bac3b9b5afa3662b52bb95d#diff-40616c4678c3fbfe07c0701505ce0567 > I deprecated some configuration keys. > They are unused and need to be removed with the 2.0 release. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (FLINK-13887) ExecutionConfig#setDefaultInputDependencyConstraint should do NotNull check on params
[ https://issues.apache.org/jira/browse/FLINK-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-13887: Assignee: Zhu Zhu > ExecutionConfig#setDefaultInputDependencyConstraint should do NotNull check > on params > - > > Key: FLINK-13887 > URL: https://issues.apache.org/jira/browse/FLINK-13887 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > > If a user invokes ExecutionConfig#setDefaultInputDependencyConstraint(null) > to set the defaultInputDependencyConstraint to be null, the scheduling > topology building will throw NPE in ExecutionGraph creating stage, causing a > master node fatal error. > Thus we need to do checkNotNull on the > ExecutionConfig#setDefaultInputDependencyConstraint param to remind users in > an early stage. > > Exception is as blow: > 2019-08-28T15:19:21.287+ ERROR > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Fatal error > occurred in the cluster entrypoint. > org.apache.flink.runtime.dispatcher.DispatcherException: Failed to take > leadership with session id 2f8f7919-a81b-4529-ad57-9789dbf07707. > at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$30(Dispatcher.java:915) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > at > org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:691) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) > at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) > at > akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) > at akka.actor.Actor.aroundReceive(Actor.scala:517) > at akka.actor.Actor.aroundReceive$(Actor.scala:515) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > at > org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36) > at >
[jira] [Closed] (FLINK-3894) Remove Program Packaging via Plans
[
https://issues.apache.org/jira/browse/FLINK-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-3894.
Resolution: Duplicate
> Remove Program Packaging via Plans
> --
>
> Key: FLINK-3894
> URL: https://issues.apache.org/jira/browse/FLINK-3894
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client
>Affects Versions: 1.0.2
>Reporter: Stephan Ewen
>Priority: Major
>
> Flink still supports a legacy way of packaging programs via implementing the
> {{Program}} interface.
> The reason for that was the support of the now removed **Record API**.
> We should remove that way to package a program as well, to reduce complexity.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-3986) Rename readFileStream from the StreamExecutionEnvironment
[ https://issues.apache.org/jira/browse/FLINK-3986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-3986: - Component/s: (was: Build System) API / DataStream > Rename readFileStream from the StreamExecutionEnvironment > - > > Key: FLINK-3986 > URL: https://issues.apache.org/jira/browse/FLINK-3986 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Reporter: Kostas Kloudas >Priority: Major > Fix For: 2.0.0 > > > The readFileStream(String filePath, long intervalMillis, WatchType watchType) > has to be renamed to readFile match the naming conventions of the rest of the > methods, or even removed. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526487220 ## CI report: * 3c6c3d69983136fb8bfa11b28a7ba783dfe61e52 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125188905) * a9d65d8fabd938d47abd801a4686cb185f2fcf68 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/125206763) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9566: [FLINK-13911][ml] Add the interface of ModelDataConverter, and several base classes that implement this interface
flinkbot edited a comment on issue #9566: [FLINK-13911][ml] Add the interface of ModelDataConverter, and several base classes that implement this interface URL: https://github.com/apache/flink/pull/9566#issuecomment-526546269 ## CI report: * 8393d74d7531c5a039cfcb486e4818d9fa603754 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/125210107) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-4714) Set task state to RUNNING after state has been restored
[
https://issues.apache.org/jira/browse/FLINK-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4714:
-
Component/s: (was: Runtime / State Backends)
(was: Runtime / Coordination)
Runtime / Task
> Set task state to RUNNING after state has been restored
> ---
>
> Key: FLINK-4714
> URL: https://issues.apache.org/jira/browse/FLINK-4714
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.2.0
>Reporter: Till Rohrmann
>Assignee: Wei-Che Wei
>Priority: Major
>
> The task state is set to {{RUNNING}} as soon as the {{Task}} is executed.
> That, however, happens before the state of the {{StreamTask}} invokable has
> been restored. As a result, the {{CheckpointCoordinator}} starts to trigger
> checkpoints even though the {{StreamTask}} is not ready.
> In order to avoid aborting checkpoints and properly start it, we should
> switch the task state to {{RUNNING}} after the state has been restored.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] zhijiangW commented on a change in pull request #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput
zhijiangW commented on a change in pull request #9483: [FLINK-13767][task]
Migrate isFinished method from AvailabilityListener to AsyncDataInput
URL: https://github.com/apache/flink/pull/9483#discussion_r319454145
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/StreamTwoInputSelectableProcessor.java
##
@@ -194,23 +193,28 @@ public boolean processInput() throws Exception {
int readingInputIndex = selectNextReadingInputIndex();
if (readingInputIndex == -1) {
- return false;
+ return InputStatus.NOTHING_AVAILABLE;
}
lastReadInputIndex = readingInputIndex;
InputStatus status;
if (readingInputIndex == 0) {
status = input1.emitNext(output1);
+ firstInputStatus = status;
} else {
status = input2.emitNext(output2);
+ secondInputStatus = status;
}
checkFinished(status, lastReadInputIndex);
if (status == InputStatus.NOTHING_AVAILABLE) {
inputSelectionHandler.setUnavailableInput(readingInputIndex);
}
- return status == InputStatus.MORE_AVAILABLE;
+ if (status == InputStatus.END_OF_INPUT && secondInputStatus !=
InputStatus.END_OF_INPUT) {
Review comment:
I think I just keep the same behavior as before. Although I also felt this
logic seem a bit tough to understand, I have not thought through whether we
need to refactor this logic beforehand.
If `status == NOTHING_AVAILABLE` and `secondInputStatus == MORE_AVAILABLE`,
it would return `status` and in `StreamTask#performDefaultAction` it would call
`inputProcessor.isAvailable().thenRun()`.
Before refactoring, if `status == NOTHING_AVAILABLE` which corresponds to
the case of returning null `recordOrMark`, then in
`StreamTask#performDefaultAction` it would still call
`inputProcessor.isAvailable().thenRun()`.
Also in previous behavior only two inputs are both finished, then we can
return `status==END_OF_INPUT`. If only current `status== END_OF_INPUT` then the
returned `recordOrMark` should be null, so in `StreamTask#performDefaultAction`
it would also enter the path of `inputProcessor.isAvailable().thenRun()`. In
order to keep this same behavior we need to return `NOTHING_AVAILABLE` here to
enter the same path in `StreamTask`.
So this part was my main concern before, whether it is worth doing this
refactor here. Because we add this condition `if (status ==
InputStatus.END_OF_INPUT && secondInputStatus != InputStatus.END_OF_INPUT)` to
always check in every call for keeping the previous behavior.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] 1u0 commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
1u0 commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319463621
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/SynchronousCheckpointITCase.java
##
@@ -151,11 +152,15 @@ protected void performDefaultAction(DefaultActionContext
context) throws Excepti
}
@Override
- public boolean triggerCheckpoint(CheckpointMetaData
checkpointMetaData, CheckpointOptions checkpointOptions, boolean
advanceToEndOfEventTime) throws Exception {
- eventQueue.put(Event.PRE_TRIGGER_CHECKPOINT);
- boolean result =
super.triggerCheckpoint(checkpointMetaData, checkpointOptions,
advanceToEndOfEventTime);
- eventQueue.put(Event.POST_TRIGGER_CHECKPOINT);
- return result;
+ public Future
triggerCheckpointAsync(CheckpointMetaData checkpointMetaData, CheckpointOptions
checkpointOptions, boolean advanceToEndOfEventTime) {
+ try {
+ eventQueue.put(Event.PRE_TRIGGER_CHECKPOINT);
+ Future result =
super.triggerCheckpointAsync(checkpointMetaData, checkpointOptions,
advanceToEndOfEventTime);
+ eventQueue.put(Event.POST_TRIGGER_CHECKPOINT);
+ return result;
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
Review comment:
This `InterruptedException` is from `LinkiedBlockingQueue.put()`
(`eventQueue.put()`).
The queue is not bounded and I hope this exception won't be thrown here
during tests runs.
But even if thrown, I'd prefer to propagate them.
**Update:** or do you mean to handle those exception here by calling
`Thread.currentThread().interrupt();`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526484775 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a9d65d8fabd938d47abd801a4686cb185f2fcf68 (Fri Aug 30 11:03:52 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-12958).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-13920) Move list of old releases into _config.yml
Chesnay Schepler created FLINK-13920: Summary: Move list of old releases into _config.yml Key: FLINK-13920 URL: https://issues.apache.org/jira/browse/FLINK-13920 Project: Flink Issue Type: Improvement Components: Project Website Reporter: Chesnay Schepler Assignee: Chesnay Schepler When adding new releases to the downloads page, some information must be updated in _config.yml (used in the upper parts of the downloads page), but the list of past releases must be updated separately on both download pages (english/chinese). This is error-prone; we should move the list of old releases into _config.yml as well. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] 1u0 commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
1u0 commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319468921
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/SystemProcessingTimeServiceTest.java
##
@@ -391,61 +384,6 @@ public void onProcessingTime(long timestamp) throws
Exception {}
}
}
- @Test
Review comment:
I find those two tests not applicable to `SystemProcessingTimeService`, as
`AsyncExceptionHandler` is not part of `SystemProcessingTimeService` api
contract anymore.
But I'll try to add something similar for
`StreamTask.TimerInvocationContext` as a test.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
flinkbot edited a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526484775 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a9d65d8fabd938d47abd801a4686cb185f2fcf68 (Fri Aug 30 11:22:13 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-12958).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] dawidwys commented on a change in pull request #9485: [FLINK-13775][table-planner-blink] Refactor ExpressionConverter(RexNodeConverter) in blink
dawidwys commented on a change in pull request #9485:
[FLINK-13775][table-planner-blink] Refactor
ExpressionConverter(RexNodeConverter) in blink
URL: https://github.com/apache/flink/pull/9485#discussion_r319381186
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/expressions/converter/DefinedConvertRule.java
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.expressions.converter;
+
+import org.apache.flink.table.expressions.CallExpression;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.table.functions.FunctionDefinition;
+import org.apache.flink.table.planner.functions.sql.FlinkSqlOperatorTable;
+
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.sql.SqlOperator;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Defined function {@link CallExpressionConvertRule}, it included conversions
are one-to-one with
+ * calcite SqlOperator.
+ */
+public class DefinedConvertRule implements CallExpressionConvertRule {
+
+ private static final Map
DEFINITION_OPERATOR_MAP = new HashMap<>();
+ static {
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.SHA1,
FlinkSqlOperatorTable.SHA1);
+
+ // logic functions
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.AND,
FlinkSqlOperatorTable.AND);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.OR,
FlinkSqlOperatorTable.OR);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.NOT,
FlinkSqlOperatorTable.NOT);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.IF,
FlinkSqlOperatorTable.CASE);
+
+ // comparison functions
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.EQUALS,
FlinkSqlOperatorTable.EQUALS);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.GREATER_THAN,
FlinkSqlOperatorTable.GREATER_THAN);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.GREATER_THAN_OR_EQUAL,
FlinkSqlOperatorTable.GREATER_THAN_OR_EQUAL);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.LESS_THAN,
FlinkSqlOperatorTable.LESS_THAN);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.LESS_THAN_OR_EQUAL,
FlinkSqlOperatorTable.LESS_THAN_OR_EQUAL);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.NOT_EQUALS,
FlinkSqlOperatorTable.NOT_EQUALS);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.IS_NULL,
FlinkSqlOperatorTable.IS_NULL);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.IS_NOT_NULL,
FlinkSqlOperatorTable.IS_NOT_NULL);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.IS_TRUE,
FlinkSqlOperatorTable.IS_TRUE);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.IS_FALSE,
FlinkSqlOperatorTable.IS_FALSE);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.IS_NOT_TRUE,
FlinkSqlOperatorTable.IS_NOT_TRUE);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.IS_NOT_FALSE,
FlinkSqlOperatorTable.IS_NOT_FALSE);
+
+ // string functions
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.CHAR_LENGTH,
FlinkSqlOperatorTable.CHAR_LENGTH);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.INIT_CAP,
FlinkSqlOperatorTable.INITCAP);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.LIKE,
FlinkSqlOperatorTable.LIKE);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.LOWER,
FlinkSqlOperatorTable.LOWER);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.SIMILAR,
FlinkSqlOperatorTable.SIMILAR_TO);
+
DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.SUBSTRING,
FlinkSqlOperatorTable.SUBSTRING);
+ DEFINITION_OPERATOR_MAP.put(BuiltInFunctionDefinitions.UPPER,
FlinkSqlOperatorTable.UPPER);
+
[GitHub] [flink] flinkbot edited a comment on issue #9354: [FLINK-13568][sql-parser] DDL create table doesn't allow STRING data …
flinkbot edited a comment on issue #9354: [FLINK-13568][sql-parser] DDL create table doesn't allow STRING data … URL: https://github.com/apache/flink/pull/9354#issuecomment-518083679 ## CI report: * 754c52de984cb476ae0442c6704219b64c68441e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/121903039) * af75fff40f4e9e57bd09403741ff1a7c63285941 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122105364) * 8eddadbdb9543c7a42cdba7c1ebe938934671e28 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122210862) * 81f5ee77a0e7bb83ce0a2b2447e45a6c364d69ea : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/13853) * 4a41ecb487a5dadaf868814b676c51a04670b76e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/122242467) * b7ef3d2f42edabc376e859f0bdc963f1f535c812 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/125186831) * 390c0c1d31cbd3ada69f6ec375dec2b788754f85 : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions URL: https://github.com/apache/flink/pull/9562#issuecomment-526200719 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4d5aa4e6bbd588fdaa1cab4b25eb6e1ed3eabf8e (Fri Aug 30 07:02:37 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
flinkbot commented on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526484775 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 3c6c3d69983136fb8bfa11b28a7ba783dfe61e52 (Fri Aug 30 07:02:39 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-12958).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9485: [FLINK-13775][table-planner-blink] Refactor ExpressionConverter(RexNodeConverter) in blink
flinkbot edited a comment on issue #9485: [FLINK-13775][table-planner-blink] Refactor ExpressionConverter(RexNodeConverter) in blink URL: https://github.com/apache/flink/pull/9485#issuecomment-522652393 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c4645a6663767d1dfcd242ea5ceb38ef838a5580 (Fri Aug 30 07:06:40 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9552: [FLINK-13884] Set default failure rate restart strategy delay to 0s
flinkbot edited a comment on issue #9552: [FLINK-13884] Set default failure rate restart strategy delay to 0s URL: https://github.com/apache/flink/pull/9552#issuecomment-525754671 ## CI report: * b78e598a216202595f6fd4748ca0860af3468668 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/124918117) * 1d10c2983f80da0d65517a9695cd473876c70ab5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125092286) * 2c935a81114634de8dbc1c87e329e3b1123572c3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125107962) * a67e92eb76efc48663bb4aa3af213b645d757bab : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125122488) * 95d0bf236135a5c0367d934ff7d41fd594c46f65 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125182666) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] 1u0 removed a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
1u0 removed a comment on issue #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#issuecomment-526486560 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-12164) JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable
[
https://issues.apache.org/jira/browse/FLINK-12164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919314#comment-16919314
]
Biao Liu commented on FLINK-12164:
--
Hi [~aljoscha],
Sorry I forget to update the progress. I'm not working on it for now. I have
abandoned my prior PR. I was planning to implement a better one (not mocking so
much) however recently I didn't find time to do so :(
If it becomes a blocker, I could postpone other things and get back to this
issue first.
> JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout is unstable
>
>
> Key: FLINK-12164
> URL: https://issues.apache.org/jira/browse/FLINK-12164
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.9.0
>Reporter: Aljoscha Krettek
>Assignee: Biao Liu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {code}
> 07:28:23.957 [ERROR] Tests run: 24, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed: 8.968 s <<< FAILURE! - in
> org.apache.flink.runtime.jobmaster.JobMasterTest
> 07:28:23.957 [ERROR]
> testJobFailureWhenTaskExecutorHeartbeatTimeout(org.apache.flink.runtime.jobmaster.JobMasterTest)
> Time elapsed: 0.177 s <<< ERROR!
> java.util.concurrent.ExecutionException: java.lang.Exception: Unknown
> TaskManager 69a7c8c18a36069ff90a1eae8ec41066
> at
> org.apache.flink.runtime.jobmaster.JobMasterTest.registerSlotsAtJobMaster(JobMasterTest.java:1746)
> at
> org.apache.flink.runtime.jobmaster.JobMasterTest.runJobFailureWhenTaskExecutorTerminatesTest(JobMasterTest.java:1670)
> at
> org.apache.flink.runtime.jobmaster.JobMasterTest.testJobFailureWhenTaskExecutorHeartbeatTimeout(JobMasterTest.java:1630)
> Caused by: java.lang.Exception: Unknown TaskManager
> 69a7c8c18a36069ff90a1eae8ec41066
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (FLINK-13910) Many serializable classes have no explicit 'serialVersionUID'
Yun Tang created FLINK-13910:
Summary: Many serializable classes have no explicit
'serialVersionUID'
Key: FLINK-13910
URL: https://issues.apache.org/jira/browse/FLINK-13910
Project: Flink
Issue Type: Bug
Reporter: Yun Tang
Attachments: SerializableNoSerialVersionUIDField
Currently, many serializable classes in Flink have no explicit
'serialVersionUID'. As [official
doc|https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization]
said, {{Serializable classes must define a Serial Version UID}}.
No 'serialVersionUID' would cause compatibility problem. Take
{{TwoPhaseCommitSinkFunction}} for example, since no explicit
'serialVersionUID' defined, after
[FLINK-10455|https://github.com/apache/flink/commit/489be82a6d93057ed4a3f9bf38ef50d01d11d96b]
introduced, its default 'serialVersionUID' has changed from
"4584405056408828651" to "4064406918549730832". In other words, if we submit a
job from Flink-1.6.3 local home to remote Flink-1.6.2 cluster with the usage of
{{TwoPhaseCommitSinkFunction}}, we would get exception like:
{code:java}
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
instantiate user function.
at
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:239)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:104)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InvalidClassException:
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction; local
class incompatible: stream classdesc serialVersionUID = 4584405056408828651,
local class serialVersionUID = 4064406918549730832
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:537)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:524)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:512)
at
org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:473)
at
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:224)
... 4 more
{code}
Similar problems existed in
{{org.apache.flink.streaming.api.operators.SimpleOperatorFactory}} which has
different 'serialVersionUID' from release-1.9 and current master branch.
IMO, we might have two options to fix this bug:
# Add explicit serialVersionUID for those classes which is identical to latest
Flink-1.9.0 release code.
# Use similar mechanism like {{FailureTolerantObjectInputStream}} in
{{InstantiationUtil}} to ignore serialVersionUID mismatch.
I have collected all production classes without serialVersionUID from latest
master branch in the attachment, which counts to 639 classes.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions URL: https://github.com/apache/flink/pull/9562#issuecomment-526200719 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4d5aa4e6bbd588fdaa1cab4b25eb6e1ed3eabf8e (Fri Aug 30 08:38:14 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
tillrohrmann commented on a change in pull request #9562: [FLINK-13898] Migrate
restart stratey config constants to ConfigOptions
URL: https://github.com/apache/flink/pull/9562#discussion_r319412315
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/RestartStrategyOptions.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.docs.ConfigGroup;
+import org.apache.flink.annotation.docs.ConfigGroups;
+import org.apache.flink.configuration.description.Description;
+
+import static org.apache.flink.configuration.description.LinkElement.link;
+import static org.apache.flink.configuration.description.TextElement.code;
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/**
+ * Config options for restart strategies.
+ */
+@PublicEvolving
+@ConfigGroups(groups = {
+ @ConfigGroup(name = "FixedDelayRestartStrategy", keyPrefix =
"restart-strategy.fixed-delay"),
+ @ConfigGroup(name = "FailureRateRestartStrategy", keyPrefix =
"restart-strategy.failure-rate")
+})
+public class RestartStrategyOptions {
+
+ public static final ConfigOption RESTART_STRATEGY =
ConfigOptions
+ .key("restart-strategy")
+ .noDefaultValue()
+ .withDescription(
+ Description.builder()
+ .text("Defines the restart strategy to use in
case of job failures.")
+ .linebreak()
+ .text("Accepted values are:")
+ .list(
+ text("%s, %s, %s: No restart
strategy.", code("none"), code("off"), code("disable")),
+ text(
+ "%s, %s: Fixed delay restart
strategy. More details can be found %s.",
+ code("fixeddelay"),
+ code("fixed-delay"),
+
link("../dev/task_failure_recovery.html#fixed-delay-restart-strategy", "here")),
+ text(
+ "%s, %s: Failure rate restart
strategy. More details can be found %s.",
+ code("failurerate"),
+ code("failure-rate"),
+
link("../dev/task_failure_recovery.html#failure-rate-restart-strategy",
"here")),
+ text(
+ "%s: Fully qualified name of %s
factory which has has a method %s.",
+
code("org.foobar.MyRestartStrategyFactoryFactory"),
+ code("RestartStrategyFactory"),
+ code("RestartStrategyFactory
createFactory(Configuration configuration)"))
+ )
+ .text(
+ "If checkpointing is disabled, the
default value is %s. " +
+ "If checkpointing is enabled,
the default value is %s with %s restart attempts.",
+ code("none"),
+ code("fixed-delay"),
+ code("Integer.MAX_VALUE"))
+ .build());
+
+ public static final ConfigOption
RESTART_STRATEGY_FIXED_DELAY_ATTEMPTS = ConfigOptions
+ .key("restart-strategy.fixed-delay.attempts")
+ .defaultValue(1)
+ .withDescription(
+ Description.builder()
+ .text(
+ "The number of times that Flink retries
the execution before the job is declared as failed when using %s. " +
+
[GitHub] [flink] aljoscha commented on issue #9565: [FLINK-12510] Use SpecificRecord.getSchema in AvroFactory
aljoscha commented on issue #9565: [FLINK-12510] Use SpecificRecord.getSchema in AvroFactory URL: https://github.com/apache/flink/pull/9565#issuecomment-526518444 R: @igalshilman could you maybe review this? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zentol commented on a change in pull request #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
zentol commented on a change in pull request #9562: [FLINK-13898] Migrate
restart stratey config constants to ConfigOptions
URL: https://github.com/apache/flink/pull/9562#discussion_r319416867
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/RestartStrategyOptions.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.docs.ConfigGroup;
+import org.apache.flink.annotation.docs.ConfigGroups;
+import org.apache.flink.configuration.description.Description;
+
+import static org.apache.flink.configuration.description.LinkElement.link;
+import static org.apache.flink.configuration.description.TextElement.code;
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/**
+ * Config options for restart strategies.
+ */
+@PublicEvolving
+@ConfigGroups(groups = {
+ @ConfigGroup(name = "FixedDelayRestartStrategy", keyPrefix =
"restart-strategy.fixed-delay"),
+ @ConfigGroup(name = "FailureRateRestartStrategy", keyPrefix =
"restart-strategy.failure-rate")
+})
+public class RestartStrategyOptions {
+
+ public static final ConfigOption RESTART_STRATEGY =
ConfigOptions
+ .key("restart-strategy")
+ .noDefaultValue()
+ .withDescription(
+ Description.builder()
+ .text("Defines the restart strategy to use in
case of job failures.")
+ .linebreak()
+ .text("Accepted values are:")
+ .list(
+ text("%s, %s, %s: No restart
strategy.", code("none"), code("off"), code("disable")),
+ text(
+ "%s, %s: Fixed delay restart
strategy. More details can be found %s.",
+ code("fixeddelay"),
+ code("fixed-delay"),
+
link("../dev/task_failure_recovery.html#fixed-delay-restart-strategy", "here")),
Review comment:
Can we just remove these links for now? There aren't any other instances of
options linking to other pages of the flink documentation.
If the link doesn't work on the chinese page the link checker will likely
throw a fit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9565: [FLINK-12501] Use SpecificRecord.getSchema in AvroFactory
flinkbot edited a comment on issue #9565: [FLINK-12501] Use SpecificRecord.getSchema in AvroFactory URL: https://github.com/apache/flink/pull/9565#issuecomment-526522446 ## CI report: * 247e51b1cba1bed7eaa64798623865e13c2a8c8b : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/125200467) * 0533cc9722194964f630a7de8b9ffd7a2dac5809 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/125202045) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-3827) Flink modules include unused dependencies
[
https://issues.apache.org/jira/browse/FLINK-3827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-3827.
Resolution: Abandoned
> Flink modules include unused dependencies
> -
>
> Key: FLINK-3827
> URL: https://issues.apache.org/jira/browse/FLINK-3827
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.1.0
>Reporter: Till Rohrmann
>Priority: Major
>
> A quick look via {{mvn dependency:analyze}} revealed that many Flink modules
> include dependencies which they don't really need. We should fix this for the
> next {{1.1}} release.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot commented on issue #9566: [FLINK-13911][ml] Add the interface of ModelDataConverter, and several base classes that implement this interface
flinkbot commented on issue #9566: [FLINK-13911][ml] Add the interface of ModelDataConverter, and several base classes that implement this interface URL: https://github.com/apache/flink/pull/9566#issuecomment-526541525 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8393d74d7531c5a039cfcb486e4818d9fa603754 (Fri Aug 30 09:55:33 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-13911).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13912) Remove ClusterClient#getClusterConnectionInfo
[
https://issues.apache.org/jira/browse/FLINK-13912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919381#comment-16919381
]
TisonKun commented on FLINK-13912:
--
If there is no more concern I volunteer to start progress.
> Remove ClusterClient#getClusterConnectionInfo
> -
>
> Key: FLINK-13912
> URL: https://issues.apache.org/jira/browse/FLINK-13912
> Project: Flink
> Issue Type: Improvement
> Components: Command Line Client, Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: TisonKun
>Priority: Major
> Fix For: 1.10.0
>
>
> As discussed in FLINK-13750, we actually doesn't need this method any more.
> All configuration needed is WebMonitor address and port in standalone HA
> mode. We can safely remove this method and replace its usages with
> {{ClusterClient#getWebInterfaceURL}}
> cc [~till.rohrmann]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (FLINK-7574) Remove unused dependencies from flink-clients
[ https://issues.apache.org/jira/browse/FLINK-7574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann closed FLINK-7574. Resolution: Abandoned > Remove unused dependencies from flink-clients > - > > Key: FLINK-7574 > URL: https://issues.apache.org/jira/browse/FLINK-7574 > Project: Flink > Issue Type: Sub-task > Components: Build System >Affects Versions: 1.3.2 > Environment: Apache Maven 3.3.9, Java version: 1.8.0_144 >Reporter: Hai Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @ > flink-clients_2.11 --- > [WARNING] Used undeclared dependencies found: > [WARNING]org.scala-lang:scala-library:jar:2.11.11:compile > [WARNING]com.data-artisans:flakka-actor_2.11:jar:2.3-custom:compile > [WARNING] Unused declared dependencies found: > [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test > [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile > [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test > [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile > [WARNING]log4j:log4j:jar:1.2.17:test > [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test > [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Closed] (FLINK-3828) Remove unused dependencies from flink-runtime
[
https://issues.apache.org/jira/browse/FLINK-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-3828.
Resolution: Abandoned
> Remove unused dependencies from flink-runtime
> -
>
> Key: FLINK-3828
> URL: https://issues.apache.org/jira/browse/FLINK-3828
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
> Environment: Apache Maven 3.3.9, Java version: 1.8.0_144
>Reporter: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {noformat}
> [INFO] --- maven-dependency-plugin:2.10:analyze (default-cli) @
> flink-runtime_2.11 ---
> [WARNING] Used undeclared dependencies found:
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]io.netty:netty:jar:3.8.0.Final:compile
> [WARNING]com.google.code.findbugs:annotations:jar:2.0.1:test
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]com.typesafe:config:jar:1.2.1:compile
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING]commons-io:commons-io:jar:2.4:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.powermock:powermock-api-support:jar:1.6.5:test
> [WARNING]javax.xml.bind:jaxb-api:jar:2.2.2:compile
> [WARNING]commons-collections:commons-collections:jar:3.2.2:compile
> [WARNING]com.fasterxml.jackson.core:jackson-annotations:jar:2.7.4:compile
> [WARNING]org.powermock:powermock-core:jar:1.6.5:test
> [WARNING]org.powermock:powermock-reflect:jar:1.6.5:test
> [WARNING] Unused declared dependencies found:
> [WARNING]com.data-artisans:flakka-slf4j_2.11:jar:2.3-custom:compile
> [WARNING]org.reflections:reflections:jar:0.9.10:test
> [WARNING]org.javassist:javassist:jar:3.18.2-GA:compile
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]com.twitter:chill_2.11:jar:0.7.4:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (FLINK-3829) Remove unused dependencies from flink-java
[
https://issues.apache.org/jira/browse/FLINK-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-3829.
Resolution: Abandoned
> Remove unused dependencies from flink-java
> --
>
> Key: FLINK-3829
> URL: https://issues.apache.org/jira/browse/FLINK-3829
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Reporter: Till Rohrmann
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {noformat}
> [WARNING] Used undeclared dependencies found:
> [WARNING]com.esotericsoftware.kryo:kryo:jar:2.24.0:compile
> [WARNING]commons-cli:commons-cli:jar:1.3.1:compile
> [WARNING]org.apache.flink:flink-metrics-core:jar:1.4-SNAPSHOT:compile
> [WARNING]org.hamcrest:hamcrest-core:jar:1.3:test
> [WARNING]org.apache.flink:flink-annotations:jar:1.4-SNAPSHOT:compile
> [WARNING] Unused declared dependencies found:
> [WARNING]org.hamcrest:hamcrest-all:jar:1.3:test
> [WARNING]org.powermock:powermock-module-junit4:jar:1.6.5:test
> [WARNING]com.google.code.findbugs:jsr305:jar:1.3.9:compile
> [WARNING]log4j:log4j:jar:1.2.17:test
> [WARNING]org.powermock:powermock-api-mockito:jar:1.6.5:test
> [WARNING]org.apache.flink:force-shading:jar:1.4-SNAPSHOT:compile
> [WARNING]org.slf4j:slf4j-log4j12:jar:1.7.7:test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (FLINK-13749) Make Flink client respect classloading policy
[ https://issues.apache.org/jira/browse/FLINK-13749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Zagrebin updated FLINK-13749: Affects Version/s: (was: 1.9.0) > Make Flink client respect classloading policy > - > > Key: FLINK-13749 > URL: https://issues.apache.org/jira/browse/FLINK-13749 > Project: Flink > Issue Type: Improvement > Components: Command Line Client, Runtime / REST >Reporter: Paul Lin >Assignee: Paul Lin >Priority: Minor > > Currently, Flink client does not respect the classloading policy and uses > hardcoded parent-first classloader, while the other components like > jobmanager and taskmanager use child-first classloader by default and respect > the classloading options. This makes the client more likely to have > dependency conflicts, especially after we removed the convenient hadoop > binaries (so users need to add hadoop classpath in the client classpath). > So I propose to make Flink client's (including cli and rest handler) > classloading behavior aligned with the other components. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (FLINK-4272) Create a JobClient for job control and monitoring
[
https://issues.apache.org/jira/browse/FLINK-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-4272:
-
Component/s: (was: Command Line Client)
> Create a JobClient for job control and monitoring
> --
>
> Key: FLINK-4272
> URL: https://issues.apache.org/jira/browse/FLINK-4272
> Project: Flink
> Issue Type: New Feature
> Components: API / DataStream
>Reporter: Maximilian Michels
>Assignee: Maximilian Michels
>Priority: Minor
>
> The aim of this new features is to expose a client to the user which allows
> to cancel a running job, retrieve accumulators for a running job, or perform
> other actions in the future. Let's call it {{JobClient}} for now (although
> this clashes with the existing JobClient class which could be renamed to
> JobClientActorUtils instead).
> The new client should be returned from the {{ClusterClient}} class upon job
> submission. The client should also be instantiatable by the users to retrieve
> the JobClient with a JobID.
> We should expose the new JobClient to the Java and Scala APIs using a new
> method on the {{ExecutionEnvironment}} / {{StreamExecutionEnvironment}}
> called {{executeWithControl()}} (perhaps we can find a better name).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (FLINK-4818) RestartStrategy should track how many failed restore attempts the same checkpoint has and fall back to earlier checkpoints
[
https://issues.apache.org/jira/browse/FLINK-4818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-4818.
Resolution: Won't Do
I guess this can now be done via the {{CheckpointFailureManager}}
> RestartStrategy should track how many failed restore attempts the same
> checkpoint has and fall back to earlier checkpoints
> --
>
> Key: FLINK-4818
> URL: https://issues.apache.org/jira/browse/FLINK-4818
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Stephan Ewen
>Priority: Major
>
> The restart strategies can use the exception information from FLINK-4816 to
> keep track of how often a checkpoint restore has failed. After a certain
> number of consecutive failures, they should take earlier completed
> checkpoints as recovery points.
> It is up to discussion whether the restart strategies are the right place to
> implement that, or whether this is an orthogonal feature that should go into
> the checkpoint coordinator (which knows how many checkpoints are available)
> or a separate class altogether.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot edited a comment on issue #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput
flinkbot edited a comment on issue #9483: [FLINK-13767][task] Migrate isFinished method from AvailabilityListener to AsyncDataInput URL: https://github.com/apache/flink/pull/9483#issuecomment-522597807 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 725be7c2226608382c67d5b3d372886be737fff6 (Fri Aug 30 10:32:17 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-11193) rockdb customize configuration Covered by jobmaster
[
https://issues.apache.org/jira/browse/FLINK-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919456#comment-16919456
]
Andrey Zagrebin commented on FLINK-11193:
-
[~aitozi] could you change the issue status to `in progress` if you are working
on it?
> rockdb customize configuration Covered by jobmaster
> ---
>
> Key: FLINK-11193
> URL: https://issues.apache.org/jira/browse/FLINK-11193
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.6.2, 1.7.0
>Reporter: Fan weiwen
>Assignee: Aitozi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> set state backends is rocksdb
> set 'state.backend.rocksdb.timer-service.factory' by Configuration
> {code:java}
> RocksDBStateBackend backend = new RocksDBStateBackend(checkpointPath,true);
> Configuration configuration = new Configuration();
> configuration.setString(
>RocksDBOptions.TIMER_SERVICE_FACTORY,
>RocksDBStateBackend.PriorityQueueStateType.ROCKSDB.toString());
> backend = backend.configure(configuration);
> {code}
> then submit the job by client
> but the job runtime the 'TIMER_SERVICE_FACTORY' Covered by jobmaster
> jobmaster code
> org.apache.flink.runtime.state.StateBackendLoader
> {code:java}
> backend = ((ConfigurableStateBackend) fromApplication).configure(config);
> {code}
> jobmaster buildGraph config is from flink-conf.yaml
> Final execution of the same method
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend
> {code:java}
> final String priorityQueueTypeString =
> config.getString(TIMER_SERVICE_FACTORY);
> this.priorityQueueStateType = priorityQueueTypeString.length() > 0 ?
>PriorityQueueStateType.valueOf(priorityQueueTypeString.toUpperCase()) :
> original.priorityQueueStateType;
> {code}
> the problem is config.getString(TIMER_SERVICE_FACTORY) always have value
> regardless of configuration
> so my customize config is Covered by jobmaster
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [flink] flinkbot commented on issue #9567: [FLINK-13887] [core] Ensure defaultInputDependencyConstraint to be non-null when setting it in ExecutionConfig
flinkbot commented on issue #9567: [FLINK-13887] [core] Ensure defaultInputDependencyConstraint to be non-null when setting it in ExecutionConfig URL: https://github.com/apache/flink/pull/9567#issuecomment-526567599 ## CI report: * 16d4196af29586277b0366fde4ee2af0251df34b : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13920) Move list of old releases into _config.yml
[ https://issues.apache.org/jira/browse/FLINK-13920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-13920: --- Labels: pull-request-available (was: ) > Move list of old releases into _config.yml > -- > > Key: FLINK-13920 > URL: https://issues.apache.org/jira/browse/FLINK-13920 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > > When adding new releases to the downloads page, some information must be > updated in _config.yml (used in the upper parts of the downloads page), but > the list of past releases must be updated separately on both download pages > (english/chinese). > This is error-prone; we should move the list of old releases into _config.yml > as well. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] pnowojski commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperat
pnowojski commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319468278
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -428,6 +429,10 @@ public final void invoke() throws Exception {
// See FLINK-7430
isRunning = false;
}
+ MailboxExecutor mainMailboxExecutor =
mailboxProcessor.getMainMailboxExecutor();
Review comment:
Also we should have a dedicated unit test for that (unless it's too
complicated to setup, but it would be better not to relay on ITCase's for this
feature).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperat
pnowojski commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319472433
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java
##
@@ -1151,7 +1151,10 @@ public void run() {
FileSystemSafetyNet.setSafetyNetCloseableRegistryForThread(safetyNetCloseableRegistry);
try {
-
invokable.triggerCheckpoint(checkpointMetaData, checkpointOptions,
advanceToEndOfEventTime);
+
invokable.triggerCheckpointAsync(checkpointMetaData, checkpointOptions,
advanceToEndOfEventTime);
Review comment:
missing `.get()`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperat
pnowojski commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319480244
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java
##
@@ -1183,8 +1183,12 @@ public void notifyCheckpointComplete(final long
checkpointID) {
@Override
public void run() {
try {
-
invokable.notifyCheckpointComplete(checkpointID);
- } catch (Throwable t) {
+
invokable.notifyCheckpointCompleteAsync(checkpointID);
Review comment:
if we do not wait, then we will miss the error handling?
But please first respond to my comment about double/nested mailbox/enqueuing
pattern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperat
pnowojski commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319467301
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
##
@@ -1368,13 +1369,19 @@ private void
checkpointStreamOperator(StreamOperator op) throws Exception {
private class TimerInvocationContext implements
SystemProcessingTimeService.ScheduledCallbackExecutionContext {
@Override
- public void invoke(ProcessingTimeCallback callback, long
timestamp) {
- synchronized (getCheckpointLock()) {
- try {
- callback.onProcessingTime(timestamp);
- } catch (Throwable t) {
- handleAsyncException("Caught exception
while processing timer.", new TimerException(t));
- }
+ public void invoke(ProcessingTimeCallback callback, long
timestamp) throws InterruptedException {
+ try {
+
mailboxProcessor.getMailboxExecutor(TaskMailbox.MAX_PRIORITY).execute(() -> {
+ synchronized (getCheckpointLock()) {
+ try {
+
callback.onProcessingTime(timestamp);
+ } catch (Throwable t) {
+
handleAsyncException("Caught exception while processing timer.", new
TimerException(t));
+ }
+ }
+ });
+ } catch (Throwable t) {
Review comment:
Unfortunately not, this is I think the upper most layer in the stack trace.
If we do not catch an exception here, it would be silently ignored. This made
the deadlock that I was debugging last week non obvious to find.
However maybe this deserves some comment here explaining this construct? Like
```
// Inner try catch handles all errors during the execution of the action in
the mailbox. Outer try catch handles errors that could happen during enqueuing
the action.
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9494: [FLINK-12847] [Connectors / Kinesis] update flink-connector-kinesis to use Apache 2.0 licensed code
flinkbot edited a comment on issue #9494: [FLINK-12847] [Connectors / Kinesis] update flink-connector-kinesis to use Apache 2.0 licensed code URL: https://github.com/apache/flink/pull/9494#issuecomment-523025067 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f047437e3886dd5e4f945ee5fb946f373ea60382 (Fri Aug 30 12:17:18 UTC 2019) **Warnings:** * **3 pom.xml files were touched**: Check for build and licensing issues. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions URL: https://github.com/apache/flink/pull/9562#issuecomment-526204724 ## CI report: * e90b46f3cfde29e5f5858b01e6ebdd4ba203393f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125089916) * 7927124fb0bf9449bd8010944abd54d4c2311196 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125107987) * 2a981df9820faff24e637e737dba4a6cda0a0764 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/125122513) * 4d5aa4e6bbd588fdaa1cab4b25eb6e1ed3eabf8e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125182685) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #8175: [FLINK-9787]Change ExecutionConfig#getGlobalJobParameters to return a…
tillrohrmann commented on a change in pull request #8175: [FLINK-9787]Change
ExecutionConfig#getGlobalJobParameters to return a…
URL: https://github.com/apache/flink/pull/8175#discussion_r319395177
##
File path:
flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java
##
@@ -1053,5 +1054,10 @@ public T getSerializer() {
public Map toMap() {
return Collections.emptyMap();
}
+
+ @Override
+ public boolean equals(Object obj) {
+ return true;
+ }
Review comment:
The `equals` method is needed for the `equals` method of the
`ExecutionConfig` which was not needed before because the field was `null`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #9336: [FLINK-13548][Deployment/YARN]Support priority of the Flink YARN application
tillrohrmann closed pull request #9336: [FLINK-13548][Deployment/YARN]Support priority of the Flink YARN application URL: https://github.com/apache/flink/pull/9336 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9563: [FLINK-13514] Fix instability in StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge
flinkbot edited a comment on issue #9563: [FLINK-13514] Fix instability in StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge URL: https://github.com/apache/flink/pull/9563#issuecomment-526217528 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 19dcda7de851bead881cfc401686469b839d8a5d (Fri Aug 30 08:05:42 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha closed pull request #9563: [FLINK-13514] Fix instability in StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge
aljoscha closed pull request #9563: [FLINK-13514] Fix instability in StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge URL: https://github.com/apache/flink/pull/9563 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] aljoscha commented on issue #9563: [FLINK-13514] Fix instability in StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge
aljoscha commented on issue #9563: [FLINK-13514] Fix instability in StreamTaskTest.testAsyncCheckpointingConcurrentCloseAfterAcknowledge URL: https://github.com/apache/flink/pull/9563#issuecomment-526503112 Merged This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
flinkbot edited a comment on issue #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions URL: https://github.com/apache/flink/pull/9562#issuecomment-526200719 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4d5aa4e6bbd588fdaa1cab4b25eb6e1ed3eabf8e (Fri Aug 30 08:34:10 UTC 2019) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #9562: [FLINK-13898] Migrate restart stratey config constants to ConfigOptions
tillrohrmann commented on a change in pull request #9562: [FLINK-13898] Migrate
restart stratey config constants to ConfigOptions
URL: https://github.com/apache/flink/pull/9562#discussion_r319411425
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/RestartStrategyOptions.java
##
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.configuration;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.docs.ConfigGroup;
+import org.apache.flink.annotation.docs.ConfigGroups;
+import org.apache.flink.configuration.description.Description;
+
+import static org.apache.flink.configuration.description.LinkElement.link;
+import static org.apache.flink.configuration.description.TextElement.code;
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/**
+ * Config options for restart strategies.
+ */
+@PublicEvolving
+@ConfigGroups(groups = {
+ @ConfigGroup(name = "FixedDelayRestartStrategy", keyPrefix =
"restart-strategy.fixed-delay"),
+ @ConfigGroup(name = "FailureRateRestartStrategy", keyPrefix =
"restart-strategy.failure-rate")
+})
+public class RestartStrategyOptions {
+
+ public static final ConfigOption RESTART_STRATEGY =
ConfigOptions
+ .key("restart-strategy")
+ .noDefaultValue()
+ .withDescription(
+ Description.builder()
+ .text("Defines the restart strategy to use in
case of job failures.")
+ .linebreak()
+ .text("Accepted values are:")
+ .list(
+ text("%s, %s, %s: No restart
strategy.", code("none"), code("off"), code("disable")),
+ text(
+ "%s, %s: Fixed delay restart
strategy. More details can be found %s.",
+ code("fixeddelay"),
+ code("fixed-delay"),
+
link("../dev/task_failure_recovery.html#fixed-delay-restart-strategy", "here")),
+ text(
+ "%s, %s: Failure rate restart
strategy. More details can be found %s.",
+ code("failurerate"),
+ code("failure-rate"),
+
link("../dev/task_failure_recovery.html#failure-rate-restart-strategy",
"here")),
+ text(
+ "%s: Fully qualified name of %s
factory which has has a method %s.",
+
code("org.foobar.MyRestartStrategyFactoryFactory"),
+ code("RestartStrategyFactory"),
+ code("RestartStrategyFactory
createFactory(Configuration configuration)"))
+ )
+ .text(
+ "If checkpointing is disabled, the
default value is %s. " +
+ "If checkpointing is enabled,
the default value is %s with %s restart attempts.",
Review comment:
Good point. I'll add it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8479: [FLINK-11193][State Backends]Use user passed configuration overriding default configuration loading from file
flinkbot commented on issue #8479: [FLINK-11193][State Backends]Use user passed configuration overriding default configuration loading from file URL: https://github.com/apache/flink/pull/8479#issuecomment-526521132 ## CI report: * fdd6f0e24a689b70d15b2dc190fc2fc3b0600bfa : UNKNOWN This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9502: [FLINK-13814][hive] HiveTableSink should strip quotes from partition …
flinkbot edited a comment on issue #9502: [FLINK-13814][hive] HiveTableSink should strip quotes from partition … URL: https://github.com/apache/flink/pull/9502#issuecomment-523778174 ## CI report: * 22278924cd49e4fab6eabb9bf4f6ec8310894808 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124170593) * 5ad14b3392b439f2b63b9fb079bf5546c874607d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/124325231) * 2215b05905a52ac58a828ce5b817481b0b132d8d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125031058) * 69012717a4b8820f9ebeb97287124aa336bdf137 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/125186844) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13910) Many serializable classes have no explicit 'serialVersionUID'
[
https://issues.apache.org/jira/browse/FLINK-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919356#comment-16919356
]
Yun Tang commented on FLINK-13910:
--
[~dawidwys] Thanks for your reply.
Actually I come across this problem when helping our customer to submit jobs
from local Flink-1.6.2 home to remote Flink-1.6.3 cluster and shocked that I
cannot submit my jobs due to the {{TwoPhaseCommitSinkFunction}} problem where
Flink-1.6.3 should keep compatibility with Flink-1.6.2. If I only create a
issue to only fix {{TwoPhaseCommitSinkFunction}}, this might not be careful
consideration. After a look up, I found many classes lack of serialVersionUID
and they would meet this problem again, that's why I create this issue.
As I have commented before, updating all 639 classes might not be a good idea,
we could limit the scope to only some specific modules. Besides, if we only
care the compatibility for classes that are annotated with {{@Public}}
annotation, we might also meet problem like {{@Internal}}
{{SimpleOperatorFactory}} which I mentioned in the description.
{code:java}
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
instantiate user function.
at
org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:250)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:115)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:370)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.InvalidClassException:
org.apache.flink.streaming.api.operators.SimpleOperatorFactory; local class
incompatible: stream classdesc serialVersionUID = -8185298274023067303, local
class serialVersionUID = -6571836910592064533
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:576)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:562)
at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:550)
at
org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:511)
at
{code}
> Many serializable classes have no explicit 'serialVersionUID'
> -
>
> Key: FLINK-13910
> URL: https://issues.apache.org/jira/browse/FLINK-13910
> Project: Flink
> Issue Type: Bug
> Components: API / Type Serialization System
>Reporter: Yun Tang
>Priority: Critical
> Fix For: 1.10.0, 1.9.1
>
> Attachments: SerializableNoSerialVersionUIDField
>
>
> Currently, many serializable classes in Flink have no explicit
> 'serialVersionUID'. As [official
> doc|https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization]
> said, {{Serializable classes must define a Serial Version UID}}.
> No 'serialVersionUID' would cause compatibility problem. Take
> {{TwoPhaseCommitSinkFunction}} for example, since no explicit
> 'serialVersionUID' defined, after
> [FLINK-10455|https://github.com/apache/flink/commit/489be82a6d93057ed4a3f9bf38ef50d01d11d96b]
> introduced, its default 'serialVersionUID' has changed from
> "4584405056408828651" to "4064406918549730832". In other words, if we submit
> a job from Flink-1.6.3 local home to remote Flink-1.6.2 cluster with the
> usage of {{TwoPhaseCommitSinkFunction}}, we would get exception like:
> {code:java}
> org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot
> instantiate user function.
> at
> org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperator(StreamConfig.java:239)
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.(OperatorChain.java:104)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:267)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException:
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction;
> local class
[GitHub] [flink] pnowojski commented on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one
pnowojski commented on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one URL: https://github.com/apache/flink/pull/9252#issuecomment-526528174 Thanks for the update :) I will probably merge it later this day (after a benchmark request run completes). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-3456) Backpressure Monitoring does not work for Kafka Sources
[ https://issues.apache.org/jira/browse/FLINK-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-3456: - Component/s: Connectors / Kafka > Backpressure Monitoring does not work for Kafka Sources > --- > > Key: FLINK-3456 > URL: https://issues.apache.org/jira/browse/FLINK-3456 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Runtime / Coordination >Affects Versions: 1.0.0 >Reporter: Stephan Ewen >Priority: Major > > For sources that run with Kafka 0.8, the main thread is not the thread that > processes elements and pushes them into the network. The Kafka fetcher > threads actually emit elements to the SourceContext. > Because the backpressure sampler samples the main thread, it does not > correctly handle Kafka Sources. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (FLINK-13887) ExecutionConfig#setDefaultInputDependencyConstraint should do NotNull check on params
[ https://issues.apache.org/jira/browse/FLINK-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919378#comment-16919378 ] Zhu Zhu commented on FLINK-13887: - [~Zentol] Yes I can fix it. > ExecutionConfig#setDefaultInputDependencyConstraint should do NotNull check > on params > - > > Key: FLINK-13887 > URL: https://issues.apache.org/jira/browse/FLINK-13887 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.9.0, 1.10.0 >Reporter: Zhu Zhu >Priority: Major > > If a user invokes ExecutionConfig#setDefaultInputDependencyConstraint(null) > to set the defaultInputDependencyConstraint to be null, the scheduling > topology building will throw NPE in ExecutionGraph creating stage, causing a > master node fatal error. > Thus we need to do checkNotNull on the > ExecutionConfig#setDefaultInputDependencyConstraint param to remind users in > an early stage. > > Exception is as blow: > 2019-08-28T15:19:21.287+ ERROR > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Fatal error > occurred in the cluster entrypoint. > org.apache.flink.runtime.dispatcher.DispatcherException: Failed to take > leadership with session id 2f8f7919-a81b-4529-ad57-9789dbf07707. > at > org.apache.flink.runtime.dispatcher.Dispatcher.lambda$null$30(Dispatcher.java:915) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) > at > org.apache.flink.runtime.concurrent.FutureUtils$WaitingConjunctFuture.handleCompletedFuture(FutureUtils.java:691) > at > java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736) > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) > at > java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:561) > at > java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:739) > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190) > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction.applyOrElse(PartialFunction.scala:123) > at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122) > at > akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172) > at akka.actor.Actor.aroundReceive(Actor.scala:517) > at akka.actor.Actor.aroundReceive$(Actor.scala:515) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.lang.RuntimeException: > org.apache.flink.runtime.client.JobExecutionException: Could not set up > JobManager > at > org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36) > at >
[jira] [Commented] (FLINK-13553) KvStateServerHandlerTest.readInboundBlocking unstable on Travis
[
https://issues.apache.org/jira/browse/FLINK-13553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919390#comment-16919390
]
Chesnay Schepler commented on FLINK-13553:
--
[~kkl0u] ping
> KvStateServerHandlerTest.readInboundBlocking unstable on Travis
> ---
>
> Key: FLINK-13553
> URL: https://issues.apache.org/jira/browse/FLINK-13553
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Queryable State
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The {{KvStateServerHandlerTest.readInboundBlocking}} and
> {{KvStateServerHandlerTest.testQueryExecutorShutDown}} fail on Travis with a
> {{TimeoutException}}.
> https://api.travis-ci.org/v3/job/566420641/log.txt
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Closed] (FLINK-4136) Add a JobHistory Server service
[ https://issues.apache.org/jira/browse/FLINK-4136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann closed FLINK-4136. Resolution: Duplicate > Add a JobHistory Server service > --- > > Key: FLINK-4136 > URL: https://issues.apache.org/jira/browse/FLINK-4136 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Reporter: Márton Balassi >Priority: Major > > When running on YARN the lack of a JobHistory server is very inconvenient. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] pnowojski edited a comment on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one
pnowojski edited a comment on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one URL: https://github.com/apache/flink/pull/9252#issuecomment-526555315 Merging. Lets monitor long term trends in those benchmarks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] pnowojski commented on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one
pnowojski commented on issue #9252: [FLINK-13051][runtime] Replace the non-selectable stream task with the input-selectable one URL: https://github.com/apache/flink/pull/9252#issuecomment-526555315 Thanks, merging. Lets monitor long term trends in those benchmarks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] 1u0 commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
1u0 commented on a change in pull request #9564:
[FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox
for timer triggers, checkpoints and AsyncWaitOperator
URL: https://github.com/apache/flink/pull/9564#discussion_r319458168
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java
##
@@ -1183,8 +1183,12 @@ public void notifyCheckpointComplete(final long
checkpointID) {
@Override
public void run() {
try {
-
invokable.notifyCheckpointComplete(checkpointID);
- } catch (Throwable t) {
+
invokable.notifyCheckpointCompleteAsync(checkpointID);
Review comment:
Correct me if I'm wrong, but I think, we don't need to wait for a result
here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Updated] (FLINK-13612) StateDescriptor Loading Error NPE at FlinkKafkaProducer011 with High Concurrency Initialization
[ https://issues.apache.org/jira/browse/FLINK-13612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-13612: - Fix Version/s: (was: shaded-7.0) > StateDescriptor Loading Error NPE at FlinkKafkaProducer011 with High > Concurrency Initialization > --- > > Key: FLINK-13612 > URL: https://issues.apache.org/jira/browse/FLINK-13612 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: shaded-7.0, 1.6.3, 1.6.4, 1.7.2 >Reporter: weiyunqing >Priority: Major > Fix For: 1.6.3, 1.6.4, 1.7.2 > > > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011#NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR > The NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR variable state in > FlinkKafkaProducer011 is modified with static > NullPointerException occur in high concurrency when > initializeSerializerUnlessSet method is executed > > java.lang.NullPointerException at > org.apache.flink.api.common.state.StateDescriptor.initializeSerializerUnlessSet(StateDescriptor.java:264) > at > org.apache.flink.runtime.state.DefaultOperatorStateBackend.getListState(DefaultOperatorStateBackend.java:730) > at > org.apache.flink.runtime.state.DefaultOperatorStateBackend.getUnionListState(DefaultOperatorStateBackend.java:271) > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:837) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178) > at > org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:281) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:730) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:295) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:720) at > java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [flink] 1u0 commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator
1u0 commented on a change in pull request #9564: [FLINK-12481][FLINK-12482][FLINK-12958] Streaming runtime: integrate mailbox for timer triggers, checkpoints and AsyncWaitOperator URL: https://github.com/apache/flink/pull/9564#discussion_r319457790 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ## @@ -27,7 +27,6 @@ import org.apache.flink.configuration.TaskManagerOptions; Review comment: Do you mean `FileSystemSafetyNet` usage? I can do. My reasoning is because the thread is now only passing execution from one thread (RPC) to another (main task thread) and not calling user code anymore. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-13919) Remove 1.6.4 from downloads page and SVN
[ https://issues.apache.org/jira/browse/FLINK-13919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919424#comment-16919424 ] Chesnay Schepler commented on FLINK-13919: -- Unless anyone objects I will do this next week. > Remove 1.6.4 from downloads page and SVN > > > Key: FLINK-13919 > URL: https://issues.apache.org/jira/browse/FLINK-13919 > Project: Flink > Issue Type: Improvement > Components: Project Website >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > > 1.6.4 is still listed on the download page, and the release artifacts are > still present on dist.apache.org . > Given that we no longer support 1.6 and the last release (1.6.4) was 6 months > ago I would say it is time to remove these. -- This message was sent by Atlassian Jira (v8.3.2#803003)
