[GitHub] [flink] flinkbot edited a comment on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
flinkbot edited a comment on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832141758 ## CI report: * f3c98471c5902ebf1ad8528235d4854c0c0a3cc0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17559) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15832: [FLINK-22494][ha] Introduce PossibleInconsistentStateException
flinkbot edited a comment on pull request #15832: URL: https://github.com/apache/flink/pull/15832#issuecomment-832214416 ## CI report: * a14a487a208ab8c5ea047665e472fd27f829e51c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
flinkbot edited a comment on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832141758 ## CI report: * f3c98471c5902ebf1ad8528235d4854c0c0a3cc0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17559) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
flinkbot edited a comment on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832141758 ## CI report: * f3c98471c5902ebf1ad8528235d4854c0c0a3cc0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17549) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17559) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] LinyuYao1021 commented on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
LinyuYao1021 commented on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832339034 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
flinkbot edited a comment on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832141758 ## CI report: * f3c98471c5902ebf1ad8528235d4854c0c0a3cc0 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15829: [FLINK-22266][hotfix] Do not clean up jobs in AbstractTestBase if the MiniCluster is not running
flinkbot edited a comment on pull request #15829: URL: https://github.com/apache/flink/pull/15829#issuecomment-832067869 ## CI report: * c3cd1869dd193119c55d514913fabb4da1e41ed7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17548) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
flinkbot edited a comment on pull request #15826: URL: https://github.com/apache/flink/pull/15826#issuecomment-831762997 ## CI report: * f2158c0730f5bfe61d13d8f0850b40059a3bfadc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17547) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] sjwiesman opened a new pull request #444: [hotfix] Link to StateBackend migration guide from the release notes
sjwiesman opened a new pull request #444: URL: https://github.com/apache/flink-web/pull/444 **DO NOT MERGE** until the after the next docs rebuild. Otherwise the link will 404 :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15832: [FLINK-22494][ha] Introduce PossibleInconsistentStateException
flinkbot edited a comment on pull request #15832: URL: https://github.com/apache/flink/pull/15832#issuecomment-832214416 ## CI report: * a14a487a208ab8c5ea047665e472fd27f829e51c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22563) Add migration guide for new StateBackend interfaces
[ https://issues.apache.org/jira/browse/FLINK-22563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman updated FLINK-22563: - Fix Version/s: 1.13.1 1.14.0 > Add migration guide for new StateBackend interfaces > --- > > Key: FLINK-22563 > URL: https://issues.apache.org/jira/browse/FLINK-22563 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.14.0, 1.13.1 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0, 1.13.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-22563) Add migration guide for new StateBackend interfaces
[ https://issues.apache.org/jira/browse/FLINK-22563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Seth Wiesman closed FLINK-22563. Resolution: Fixed > Add migration guide for new StateBackend interfaces > --- > > Key: FLINK-22563 > URL: https://issues.apache.org/jira/browse/FLINK-22563 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.14.0, 1.13.1 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22563) Add migration guide for new StateBackend interfaces
[ https://issues.apache.org/jira/browse/FLINK-22563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339285#comment-17339285 ] Seth Wiesman commented on FLINK-22563: -- fixed in 1.13: ff1005524641dc5cdcb8188aa9528f6c1a4b92a8 fixed in master: f6d93e85cf6ef6db88bab67557fc88b46bde69e1 > Add migration guide for new StateBackend interfaces > --- > > Key: FLINK-22563 > URL: https://issues.apache.org/jira/browse/FLINK-22563 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.14.0, 1.13.1 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman closed pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
sjwiesman closed pull request #15831: URL: https://github.com/apache/flink/pull/15831 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15832: [FLINK-22494][ha] Introduce PossibleInconsistentStateException
flinkbot commented on pull request #15832: URL: https://github.com/apache/flink/pull/15832#issuecomment-832214416 ## CI report: * a14a487a208ab8c5ea047665e472fd27f829e51c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
flinkbot edited a comment on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832154834 ## CI report: * a53ddc5b667675d973e9b868616693137860754c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17550) * 81e516edc2e4bdcea362ba97abee2eb3c2845ad9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17551) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] NicoK commented on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
NicoK commented on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832207746 @flinkbot approve all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15832: [FLINK-22494][ha] Introduce PossibleInconsistentStateException
flinkbot commented on pull request #15832: URL: https://github.com/apache/flink/pull/15832#issuecomment-832197189 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a14a487a208ab8c5ea047665e472fd27f829e51c (Tue May 04 19:44:16 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22494) Avoid discarding checkpoints in case of failure
[
https://issues.apache.org/jira/browse/FLINK-22494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-22494:
---
Labels: pull-request-available (was: )
> Avoid discarding checkpoints in case of failure
> ---
>
> Key: FLINK-22494
> URL: https://issues.apache.org/jira/browse/FLINK-22494
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.0, 1.14.0, 1.12.3
>Reporter: Matthias
>Assignee: Matthias
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.1, 1.12.4
>
>
> Both {{StateHandleStore}} implementations (i.e.
> [KubernetesStateHandleStore:157|https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/highavailability/KubernetesStateHandleStore.java#L157]
> and
> [ZooKeeperStateHandleStore:170|https://github.com/apache/flink/blob/c6997c97c575d334679915c328792b8a3067cfb5/flink-runtime/src/main/java/org/apache/flink/runtime/zookeeper/ZooKeeperStateHandleStore.java#L170])
> discard checkpoints if the checkpoint metadata wasn't written to the
> backend.
> This does not cover the cases where the data was actually written to the
> backend but the call failed anyway (e.g. due to network issues). In such a
> case, we might end up having a pointer in the backend pointing to a
> checkpoint that was discarded.
> Instead of discarding the checkpoint data in this case, we might want to keep
> it for this specific use case. Otherwise, we might run into Exceptions when
> recovering from the Checkpoint later on. We might want to add a warning to
> the user pointing to the possibly orphaned checkpoint data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] XComp opened a new pull request #15832: [FLINK-22494][ha] Introduce PossibleInconsistentStateException
XComp opened a new pull request #15832: URL: https://github.com/apache/flink/pull/15832 ## What is the purpose of the change Updating the Checkpoint Metadata store of ZooKeeper and Kubernetes might lead into situations where the data was actually written but the request still failed (due to network issue for instance). In these cases, we end up in an inconsistent state where a reference exists in ZooKeeper or the k8s ConfigMap but the corresponding Checkpoint's state is discarded due to the failure. That might cause problems during recovery. ## Brief change log * Introduces `PossibleInconsistentStateException` * `Fabric8FlinkKubeClient` translates network issues into `PossibleInconsistentStateException` * `StateHandleStore` and its implementing classes handle and forward `PossibleInconsistentStateException` * Extends `TestingLongStateHandleHelper` to work with reference instead of saving the state in the handle itself. The previous implementation caused problems when deserializing the StateHandles not referencing the actual data but providing a copy of the state. * `CheckpointCoordinator` handles `PossibleInconsistentStateException` ## Verifying this change The following tests were extended: * `CheckpointCoordinatorFailureTest` checks the failure handling for `PossibleInconsistentStateException` and another exception * `KubernetesStateHandleStoreTest` and `ZooKeeperStateHandleStoreTest` were extended to cover the relevant exceptions * `Fabric8FlinkKubeClientTest` was extended to cover both failure cases ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
flinkbot edited a comment on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832154834 ## CI report: * a53ddc5b667675d973e9b868616693137860754c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17550) * 81e516edc2e4bdcea362ba97abee2eb3c2845ad9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
flinkbot edited a comment on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832154834 ## CI report: * a53ddc5b667675d973e9b868616693137860754c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17550) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sjwiesman commented on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
sjwiesman commented on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832165703 Thanks for the fast review @NicoK, I've applied your changes to both files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] NicoK commented on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
NicoK commented on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832157337 @flinkbot approve-until architecture -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] NicoK commented on a change in pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
NicoK commented on a change in pull request #15831:
URL: https://github.com/apache/flink/pull/15831#discussion_r626011160
##
File path: docs/content.zh/docs/ops/state/state_backends.md
##
@@ -331,3 +331,132 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
```
{{< top >}}
+
+## Migrating from Legacy Backends
+
+Beginning in **Flink 1.**13**, the community reworked its public state backend
classes to help users better understand the separation of local state storage
and checkpoint storage.
Review comment:
```suggestion
Beginning in **Flink 1.13**, the community reworked its public state backend
classes to help users better understand the separation of local state storage
and checkpoint storage.
```
##
File path: docs/content.zh/docs/ops/state/state_backends.md
##
@@ -331,3 +331,132 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
```
{{< top >}}
+
+## Migrating from Legacy Backends
+
+Beginning in **Flink 1.**13**, the community reworked its public state backend
classes to help users better understand the separation of local state storage
and checkpoint storage.
+This change does not affect runtime implementation or characteristics of
Flink's state backend or checkpointing process; it is simply to communicate
intent better.
+Users can migrate existing applications to use the new API without losing any
state or consistency.
+
+### MemoryStateBackend
+
+The legacy `MemoryStateBackend` is equivalent to using
[`HashMapStateBackend`](#the-hashmapstatebackend) and
[`JobManagerCheckpointStorage`]({{< ref
"docs/ops/state/checkpoints#the-jobmanagercheckpointstorage" >}}).
+
+ `flink-conf.yaml` configuration
+
+```yaml
+state.backend: hashmap
+
+# Optional, Flink will automatically default to JobManagerCheckpointStorage
+# when no checkpoint directory is specified.
+state.checkpoint-storage: jobmanager
+```
+
+ Code Configuration
+
+{{< tabs "memorystatebackendmigration" >}}
+{{< tab "Java" >}}
+```java
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+env.setStateBackend(new HashMapStateBackend());
+env.getCheckpointConfig().setCheckpointStorage(new JobManagerStateBackend());
+```
+{{< /tab >}}
+{{< tab "Scala" >}}
+```scala
+val env = StreamExecutionEnvironment.getExecutionEnvironment
+env.setStateBackend(new HashMapStateBackend)
+env.getCheckpointConfig().setCheckpointStorage(new JobManagerStateBackend)
+```
+{{< /tab >}}
+{{< /tabs>}}
+
+### FsStateBackend
+
+The legacy `FsStateBackend` is equivalent to using
[`HashMapStateBackend`](#the-hashmapstatebackend) and
[`FileSystemCheckpointStorage`]({{< ref
"docs/ops/state/checkpoints#the-filesystemcheckpointstorage" >}}).
+
+ `flink-conf.yaml` configuration
+
+```yaml
+state.backend: hashmap
+state.checkpoints.dir: file:///checkpoint-dir/
+
+# Optional, Flink will automatically default to FileSystemCheckpointStorage
+# when a checkpoint directory is specified.
+state.checkpoint-storage: filesystem
+```
+
+ Code Configuration
+
+{{< tabs "memorystatebackendmigration" >}}
+{{< tab "Java" >}}
+```java
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+env.setStateBackend(new HashMapStateBackend());
+env.getCheckpointConfig().setCheckpointStorage("file:///checkpoint-dir");
+
+
+// Advanced FsStateBackend configurations, such as write buffer size
+// can be set by using manually instantiating a FileSystemCheckpointStorage
object.
Review comment:
```suggestion
// can be set by manually instantiating a FileSystemCheckpointStorage object.
```
##
File path: docs/content.zh/docs/ops/state/state_backends.md
##
@@ -331,3 +331,132 @@ public class MyOptionsFactory implements
ConfigurableRocksDBOptionsFactory {
```
{{< top >}}
+
+## Migrating from Legacy Backends
+
+Beginning in **Flink 1.**13**, the community reworked its public state backend
classes to help users better understand the separation of local state storage
and checkpoint storage.
+This change does not affect runtime implementation or characteristics of
Flink's state backend or checkpointing process; it is simply to communicate
intent better.
Review comment:
```suggestion
This change does not affect the runtime implementation or characteristics of
Flink's state backend or checkpointing process; it is simply to communicate
intent better.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
flinkbot commented on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832154834 ## CI report: * a53ddc5b667675d973e9b868616693137860754c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
flinkbot edited a comment on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832141758 ## CI report: * f3c98471c5902ebf1ad8528235d4854c0c0a3cc0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
flinkbot commented on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832141758 ## CI report: * f3c98471c5902ebf1ad8528235d4854c0c0a3cc0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
flinkbot commented on pull request #15831: URL: https://github.com/apache/flink/pull/15831#issuecomment-832141375 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a53ddc5b667675d973e9b868616693137860754c (Tue May 04 18:11:20 UTC 2021) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15728: [FLINK-22379][runtime] CheckpointCoordinator checks the state of all …
flinkbot edited a comment on pull request #15728: URL: https://github.com/apache/flink/pull/15728#issuecomment-824960796 ## CI report: * 87072bdee1da392386c3bd26b740f980ffee4017 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17544) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] sjwiesman commented on pull request #232: [hotfix] fix invalid module configuration
sjwiesman commented on pull request #232: URL: https://github.com/apache/flink-statefun/pull/232#issuecomment-832141007 Thanks! Merged into master and release-3.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] sjwiesman closed pull request #232: [hotfix] fix invalid module configuration
sjwiesman closed pull request #232: URL: https://github.com/apache/flink-statefun/pull/232 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] sjwiesman opened a new pull request #232: [hotfix] fix invalid module configuration
sjwiesman opened a new pull request #232: URL: https://github.com/apache/flink-statefun/pull/232 Fixes an invalid configuration and also some nits I'd been meaning to get to; mainly a misspelling and adding a full configuration example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22563) Add migration guide for new StateBackend interfaces
[ https://issues.apache.org/jira/browse/FLINK-22563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-22563: --- Labels: pull-request-available (was: ) > Add migration guide for new StateBackend interfaces > --- > > Key: FLINK-22563 > URL: https://issues.apache.org/jira/browse/FLINK-22563 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.14.0, 1.13.1 >Reporter: Seth Wiesman >Assignee: Seth Wiesman >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] sjwiesman opened a new pull request #15831: [FLINK-22563][docs] Add migration guide for new StateBackend interfaces
sjwiesman opened a new pull request #15831: URL: https://github.com/apache/flink/pull/15831 ## What is the purpose of the change Adds a migration guide to using the new state backend interfaces. ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: No ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / **docs** / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22563) Add migration guide for new StateBackend interfaces
Seth Wiesman created FLINK-22563: Summary: Add migration guide for new StateBackend interfaces Key: FLINK-22563 URL: https://issues.apache.org/jira/browse/FLINK-22563 Project: Flink Issue Type: Improvement Affects Versions: 1.14.0, 1.13.1 Reporter: Seth Wiesman Assignee: Seth Wiesman -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
flinkbot commented on pull request #15830: URL: https://github.com/apache/flink/pull/15830#issuecomment-832123926 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit f3c98471c5902ebf1ad8528235d4854c0c0a3cc0 (Tue May 04 17:43:56 UTC 2021) **Warnings:** * **1 pom.xml files were touched**: Check for build and licensing issues. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22515) Add Documentation for Flink Glue Schema Registry Integration
[ https://issues.apache.org/jira/browse/FLINK-22515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-22515: --- Labels: Glue aws pull-request-available (was: Glue aws) > Add Documentation for Flink Glue Schema Registry Integration > > > Key: FLINK-22515 > URL: https://issues.apache.org/jira/browse/FLINK-22515 > Project: Flink > Issue Type: New Feature > Components: Documentation, Formats (JSON, Avro, Parquet, ORC, > SequenceFile) >Reporter: Linyu Yao >Assignee: Linyu Yao >Priority: Major > Labels: Glue, aws, pull-request-available > > Add documentation for Flink Glue Schema Registry integration to the page of > Kafka Connector and Kinesis Connector: > [https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/kafka.html#apache-kafka-connector] > [https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/connectors/kinesis.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] LinyuYao1021 opened a new pull request #15830: [FLINK-22515][docs] Add documentation for GSR-Flink Integration
LinyuYao1021 opened a new pull request #15830: URL: https://github.com/apache/flink/pull/15830 ## What is the purpose of the change This pull request adds documentation for Glue Schema Registry(GSR)-Flink integration and updates with the latest GSR version. ## Brief change log - Documentation of Flink-Kafka and Flink-Kinesis is updated to introduce GSR integration - GSR version is updated to 1.0.2 - Using `avro` plugin to converter `.asvc` file to Java POJO class. ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22559) UpsertKafkaTableITCase fails with output mismatch
[
https://issues.apache.org/jira/browse/FLINK-22559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339164#comment-17339164
]
Roman Khachatryan commented on FLINK-22559:
---
https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=844=logs=d543d572-9428-5803-a30c-e8e09bf70915=18a8058f-13c3-5857-f389-32cb0ee00ed2=6682
> UpsertKafkaTableITCase fails with output mismatch
> -
>
> Key: FLINK-22559
> URL: https://issues.apache.org/jira/browse/FLINK-22559
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Ecosystem
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17527=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=7009
> {code}
> May 03 23:12:46 Expected: <1,name 1,2020-03-08T13:12:11.123,100,payload 1>
> May 03 23:12:46 <2,name 2,2020-03-09T13:12:11.123,101,payload 2>
> May 03 23:12:46 <3,name 3,2020-03-10T13:12:11.123,102,payload 3>
> May 03 23:12:46 <1,name 1,2020-03-08T13:12:11.123,100,payload 1>
> May 03 23:12:46 <1,name 1,2020-03-11T13:12:11.123,100,payload>
> May 03 23:12:46 but: was <1,name 1,2020-03-11T13:12:11.123,100,payload>
> May 03 23:12:46 <1,name 1,2020-03-11T13:12:11.123,100,payload>
> May 03 23:12:46 <1,name 1,2020-03-08T13:12:11.123,100,payload 1>
> May 03 23:12:46 <3,name 3,2020-03-10T13:12:11.123,102,payload 3>
> May 03 23:12:46 <2,name 2,2020-03-09T13:12:11.123,101,payload 2>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339140#comment-17339140
]
Till Rohrmann commented on FLINK-22266:
---
Thanks for the analysis [~dwysakowicz]. [~rmetzger] looking at the
{{Executing}} state, I think the current code of the {{StopWithSavepoint}}
cannot work. If the savepoint fails, we cannot go to the {{Executing}} state
because the current contract is that when entering the {{Executing}} state we
deploy the given {{ExecutionGraph}}. In this case here, the {{ExecutionGraph}}
is already running.
So I see two solutions:
1. We go into {{Restarting}} in order to restart the {{ExecutionGraph}}
2. We move the {{ExecutionGraph.deploy}} call out of the {{Executing}} state.
> Harden JobMasterStopWithSavepointITCase
> ---
>
> Key: FLINK-22266
> URL: https://issues.apache.org/jira/browse/FLINK-22266
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16451=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=3884
> {code:java}
> [ERROR]
> throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
> Time elapsed: 0.154 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithoutRestartsHelper(JobMasterStopWithSavepointITCase.java:154)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(JobMasterStopWithSavepointITCase.java:138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15829: [FLINK-22266][hotfix] Do not clean up jobs in AbstractTestBase if the MiniCluster is not running
flinkbot edited a comment on pull request #15829: URL: https://github.com/apache/flink/pull/15829#issuecomment-832067869 ## CI report: * c3cd1869dd193119c55d514913fabb4da1e41ed7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17548) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
flinkbot edited a comment on pull request #15826: URL: https://github.com/apache/flink/pull/15826#issuecomment-831762997 ## CI report: * ebdf6f053c6e8c2b07e40af8629a2a5954e37b10 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17545) * f2158c0730f5bfe61d13d8f0850b40059a3bfadc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17547) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
AHeise commented on a change in pull request #15826:
URL: https://github.com/apache/flink/pull/15826#discussion_r625931801
##
File path:
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisITCase.java
##
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.SimpleStringSchema;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.kinesis.config.AWSConfigConstants;
+import
org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.InitialPosition;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.function.RunnableWithException;
+
+import com.amazonaws.services.kinesis.model.PutRecordsRequest;
+import com.amazonaws.services.kinesis.model.PutRecordsRequestEntry;
+import org.junit.After;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.testcontainers.utility.DockerImageName;
+
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinTask;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static
org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_INITIAL_POSITION;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertThat;
+
+/** IT cases for using Kinesis consumer/producer based on Kinesalite. */
+public class FlinkKinesisITCase {
Review comment:
I double-checked and the existing test is indeed an e2e (using Flink in
docker). The new test is an ITCase (using Flink in lib).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] AHeise commented on a change in pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
AHeise commented on a change in pull request #15826:
URL: https://github.com/apache/flink/pull/15826#discussion_r625931365
##
File path:
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/KinesaliteContainer.java
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.kinesis.AmazonKinesis;
+import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
+import org.rnorth.ducttape.unreliables.Unreliables;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.wait.strategy.AbstractWaitStrategy;
+import org.testcontainers.utility.DockerImageName;
+
+import java.util.concurrent.TimeUnit;
+
+import static com.amazonaws.SDKGlobalConfiguration.ACCESS_KEY_ENV_VAR;
+import static com.amazonaws.SDKGlobalConfiguration.SECRET_KEY_ENV_VAR;
+
+/**
+ * A testcontainer based on Kinesalite.
+ *
+ * Note that the more obvious localstack container with Kinesis took 1
minute to start vs 10
+ * seconds of Kinesalite.
+ */
+public class KinesaliteContainer extends GenericContainer
{
Review comment:
I moved that to the kinesis module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15829: [FLINK-22266][hotfix] Do not clean up jobs in AbstractTestBase if the MiniCluster is not running
flinkbot commented on pull request #15829: URL: https://github.com/apache/flink/pull/15829#issuecomment-832067869 ## CI report: * c3cd1869dd193119c55d514913fabb4da1e41ed7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
flinkbot edited a comment on pull request #15826: URL: https://github.com/apache/flink/pull/15826#issuecomment-831762997 ## CI report: * ebdf6f053c6e8c2b07e40af8629a2a5954e37b10 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17545) * f2158c0730f5bfe61d13d8f0850b40059a3bfadc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] dawidwys edited a comment on pull request #443: [hotfix] Update coding guidelines for JUnit timeouts
dawidwys edited a comment on pull request #443: URL: https://github.com/apache/flink-web/pull/443#issuecomment-832057799 cc @AHeise @tillrohrmann @rmetzger @lindong28 @pnowojski -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] dawidwys commented on pull request #443: [hotfix] Update coding guidelines for JUnit timeouts
dawidwys commented on pull request #443: URL: https://github.com/apache/flink-web/pull/443#issuecomment-832057799 cc @AHeise @tillrohrmann @rmetzger @pnowojski @lindong28 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] dawidwys opened a new pull request #443: [hotfix] Update coding guidelines for JUnit timeouts
dawidwys opened a new pull request #443: URL: https://github.com/apache/flink-web/pull/443 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-22173) UnalignedCheckpointRescaleITCase fails on azure
[
https://issues.apache.org/jira/browse/FLINK-22173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-22173.
--
> UnalignedCheckpointRescaleITCase fails on azure
> ---
>
> Key: FLINK-22173
> URL: https://issues.apache.org/jira/browse/FLINK-22173
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16232=logs=d8d26c26-7ec2-5ed2-772e-7a1a1eb8317c=be5fb08e-1ad7-563c-4f1a-a97ad4ce4865=9628
> {code}
> 2021-04-08T23:25:56.3131361Z [ERROR] Tests run: 31, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 839.623 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase
> 2021-04-08T23:25:56.3132784Z [ERROR] shouldRescaleUnalignedCheckpoint[no
> scale union from 7 to
> 7](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 607.467 s <<< ERROR!
> 2021-04-08T23:25:56.3133586Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-04-08T23:25:56.3134070Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-04-08T23:25:56.3134643Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:168)
> 2021-04-08T23:25:56.3135577Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
> 2021-04-08T23:25:56.3138843Z at
> sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
> 2021-04-08T23:25:56.3139402Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-04-08T23:25:56.3139880Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-04-08T23:25:56.3140328Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-04-08T23:25:56.3140844Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-04-08T23:25:56.3141768Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-04-08T23:25:56.3142272Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-04-08T23:25:56.3142706Z at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-04-08T23:25:56.3143142Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-08T23:25:56.3143608Z at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-04-08T23:25:56.3144039Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-04-08T23:25:56.3144434Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-04-08T23:25:56.3145027Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-04-08T23:25:56.3145484Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-04-08T23:25:56.3145981Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-04-08T23:25:56.3146421Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-04-08T23:25:56.3146843Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-04-08T23:25:56.3147274Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-04-08T23:25:56.3147692Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-04-08T23:25:56.3148116Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-04-08T23:25:56.3148543Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-04-08T23:25:56.3148930Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-04-08T23:25:56.3149298Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2021-04-08T23:25:56.3149663Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-04-08T23:25:56.3150075Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-04-08T23:25:56.3150488Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-04-08T23:25:56.3151148Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-04-08T23:25:56.3151691Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-04-08T23:25:56.3152115Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-08T23:25:56.3152534Z at
>
[jira] [Updated] (FLINK-20383) DataSet allround end-to-end test fails with NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-20383:
---
Affects Version/s: 1.11.3
> DataSet allround end-to-end test fails with NullPointerException
>
>
> Key: FLINK-20383
> URL: https://issues.apache.org/jira/browse/FLINK-20383
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
>Affects Versions: 1.11.3, 1.12.0, 1.13.0
>Reporter: Robert Metzger
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.4, 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10204=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> 2020-11-26T19:45:51.5478354Z Nov 26 19:45:51 Running 'DataSet allround
> end-to-end test'
> 2020-11-26T19:45:51.5478763Z Nov 26 19:45:51
> ==
> 2020-11-26T19:45:51.5490922Z Nov 26 19:45:51 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-51548553152
> 2020-11-26T19:45:51.7135919Z Nov 26 19:45:51 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.12-SNAPSHOT-bin/flink-1.12-SNAPSHOT
> 2020-11-26T19:45:51.7232126Z Nov 26 19:45:51 Run DataSet-Allround-Test Program
> 2020-11-26T19:45:51.7266761Z Nov 26 19:45:51 Setting up SSL with: internal
> JDK dynamic
> 2020-11-26T19:45:51.7300820Z Nov 26 19:45:51 Using SAN
> dns:fv-az679-746.omrnvntobknexcvwb1hiviload.fx.internal.cloudapp.net,ip:10.1.0.4,ip:172.17.0.1,ip:172.18.0.1
> 2020-11-26T19:45:52.8564809Z Certificate was added to keystore
> 2020-11-26T19:45:54.9192018Z Certificate was added to keystore
> 2020-11-26T19:45:55.5263075Z Certificate reply was installed in keystore
> 2020-11-26T19:45:55.7151814Z Nov 26 19:45:55 Setting up SSL with: rest JDK
> dynamic
> 2020-11-26T19:45:55.7186828Z Nov 26 19:45:55 Using SAN
> dns:fv-az679-746.omrnvntobknexcvwb1hiviload.fx.internal.cloudapp.net,ip:10.1.0.4,ip:172.17.0.1,ip:172.18.0.1
> 2020-11-26T19:45:56.8610453Z Certificate was added to keystore
> 2020-11-26T19:45:59.2956135Z Certificate was added to keystore
> 2020-11-26T19:45:59.9115744Z Certificate reply was installed in keystore
> 2020-11-26T19:46:00.0800993Z Nov 26 19:46:00 Mutual ssl auth: false
> 2020-11-26T19:46:00.1688426Z Nov 26 19:46:00 Starting cluster.
> 2020-11-26T19:46:00.8775396Z Nov 26 19:46:00 Starting standalonesession
> daemon on host fv-az679-746.
> 2020-11-26T19:46:02.4016699Z Nov 26 19:46:02 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:02.4595285Z Nov 26 19:46:02 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:03.5161706Z Nov 26 19:46:03 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:04.6157725Z Nov 26 19:46:04 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:05.7153907Z Nov 26 19:46:05 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:07.4507631Z Nov 26 19:46:07 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:08.6436701Z Nov 26 19:46:08 Dispatcher REST endpoint is up.
> 2020-11-26T19:46:08.6437477Z Nov 26 19:46:08 Start 3 more task managers
> 2020-11-26T19:46:09.4234133Z Nov 26 19:46:09 [INFO] 1 instance(s) of
> taskexecutor are already running on fv-az679-746.
> 2020-11-26T19:46:09.4242548Z Nov 26 19:46:09 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:11.0906754Z Nov 26 19:46:11 [INFO] 2 instance(s) of
> taskexecutor are already running on fv-az679-746.
> 2020-11-26T19:46:11.0920876Z Nov 26 19:46:11 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:14.2637898Z Nov 26 19:46:14 [INFO] 3 instance(s) of
> taskexecutor are already running on fv-az679-746.
> 2020-11-26T19:46:14.2688442Z Nov 26 19:46:14 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:25.7685789Z Nov 26 19:46:25 Job has been submitted with
> JobID fa6775cf8a2ed83f2e4ba98930bc2cb2
> 2020-11-26T19:47:37.212Z
> 2020-11-26T19:47:37.4448196Z
>
> 2020-11-26T19:47:37.4448668Z The program finished with the following
> exception:
> 2020-11-26T19:47:37.4448911Z
> 2020-11-26T19:47:37.4453074Z
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: fa6775cf8a2ed83f2e4ba98930bc2cb2)
> 2020-11-26T19:47:37.4458530Z at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:330)
> 2020-11-26T19:47:37.4459217Z at
>
[jira] [Closed] (FLINK-20383) DataSet allround end-to-end test fails with NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-20383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-20383.
--
Fix Version/s: 1.11.4
Resolution: Fixed
back ported as 3fc8a2e into apache:release-1.11
> DataSet allround end-to-end test fails with NullPointerException
>
>
> Key: FLINK-20383
> URL: https://issues.apache.org/jira/browse/FLINK-20383
> Project: Flink
> Issue Type: Bug
> Components: API / DataSet
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Robert Metzger
>Assignee: Piotr Nowojski
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.4, 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10204=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> 2020-11-26T19:45:51.5478354Z Nov 26 19:45:51 Running 'DataSet allround
> end-to-end test'
> 2020-11-26T19:45:51.5478763Z Nov 26 19:45:51
> ==
> 2020-11-26T19:45:51.5490922Z Nov 26 19:45:51 TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-51548553152
> 2020-11-26T19:45:51.7135919Z Nov 26 19:45:51 Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.12-SNAPSHOT-bin/flink-1.12-SNAPSHOT
> 2020-11-26T19:45:51.7232126Z Nov 26 19:45:51 Run DataSet-Allround-Test Program
> 2020-11-26T19:45:51.7266761Z Nov 26 19:45:51 Setting up SSL with: internal
> JDK dynamic
> 2020-11-26T19:45:51.7300820Z Nov 26 19:45:51 Using SAN
> dns:fv-az679-746.omrnvntobknexcvwb1hiviload.fx.internal.cloudapp.net,ip:10.1.0.4,ip:172.17.0.1,ip:172.18.0.1
> 2020-11-26T19:45:52.8564809Z Certificate was added to keystore
> 2020-11-26T19:45:54.9192018Z Certificate was added to keystore
> 2020-11-26T19:45:55.5263075Z Certificate reply was installed in keystore
> 2020-11-26T19:45:55.7151814Z Nov 26 19:45:55 Setting up SSL with: rest JDK
> dynamic
> 2020-11-26T19:45:55.7186828Z Nov 26 19:45:55 Using SAN
> dns:fv-az679-746.omrnvntobknexcvwb1hiviload.fx.internal.cloudapp.net,ip:10.1.0.4,ip:172.17.0.1,ip:172.18.0.1
> 2020-11-26T19:45:56.8610453Z Certificate was added to keystore
> 2020-11-26T19:45:59.2956135Z Certificate was added to keystore
> 2020-11-26T19:45:59.9115744Z Certificate reply was installed in keystore
> 2020-11-26T19:46:00.0800993Z Nov 26 19:46:00 Mutual ssl auth: false
> 2020-11-26T19:46:00.1688426Z Nov 26 19:46:00 Starting cluster.
> 2020-11-26T19:46:00.8775396Z Nov 26 19:46:00 Starting standalonesession
> daemon on host fv-az679-746.
> 2020-11-26T19:46:02.4016699Z Nov 26 19:46:02 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:02.4595285Z Nov 26 19:46:02 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:03.5161706Z Nov 26 19:46:03 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:04.6157725Z Nov 26 19:46:04 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:05.7153907Z Nov 26 19:46:05 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:07.4507631Z Nov 26 19:46:07 Waiting for Dispatcher REST
> endpoint to come up...
> 2020-11-26T19:46:08.6436701Z Nov 26 19:46:08 Dispatcher REST endpoint is up.
> 2020-11-26T19:46:08.6437477Z Nov 26 19:46:08 Start 3 more task managers
> 2020-11-26T19:46:09.4234133Z Nov 26 19:46:09 [INFO] 1 instance(s) of
> taskexecutor are already running on fv-az679-746.
> 2020-11-26T19:46:09.4242548Z Nov 26 19:46:09 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:11.0906754Z Nov 26 19:46:11 [INFO] 2 instance(s) of
> taskexecutor are already running on fv-az679-746.
> 2020-11-26T19:46:11.0920876Z Nov 26 19:46:11 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:14.2637898Z Nov 26 19:46:14 [INFO] 3 instance(s) of
> taskexecutor are already running on fv-az679-746.
> 2020-11-26T19:46:14.2688442Z Nov 26 19:46:14 Starting taskexecutor daemon on
> host fv-az679-746.
> 2020-11-26T19:46:25.7685789Z Nov 26 19:46:25 Job has been submitted with
> JobID fa6775cf8a2ed83f2e4ba98930bc2cb2
> 2020-11-26T19:47:37.212Z
> 2020-11-26T19:47:37.4448196Z
>
> 2020-11-26T19:47:37.4448668Z The program finished with the following
> exception:
> 2020-11-26T19:47:37.4448911Z
> 2020-11-26T19:47:37.4453074Z
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: org.apache.flink.client.program.ProgramInvocationException:
> Job failed (JobID: fa6775cf8a2ed83f2e4ba98930bc2cb2)
> 2020-11-26T19:47:37.4458530Z at
>
[GitHub] [flink] pnowojski merged pull request #15824: (1.11) [FLINK-20383][runtime] Fix race condition in notification.
pnowojski merged pull request #15824: URL: https://github.com/apache/flink/pull/15824 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15829: [FLINK-22266][hotfix] Do not clean up jobs in AbstractTestBase if the MiniCluster is not running
flinkbot commented on pull request #15829: URL: https://github.com/apache/flink/pull/15829#issuecomment-832049101 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c3cd1869dd193119c55d514913fabb4da1e41ed7 (Tue May 04 15:52:09 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
flinkbot edited a comment on pull request #15826: URL: https://github.com/apache/flink/pull/15826#issuecomment-831762997 ## CI report: * ebdf6f053c6e8c2b07e40af8629a2a5954e37b10 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17545) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-22266:
---
Labels: pull-request-available test-stability (was: test-stability)
> Harden JobMasterStopWithSavepointITCase
> ---
>
> Key: FLINK-22266
> URL: https://issues.apache.org/jira/browse/FLINK-22266
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16451=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=3884
> {code:java}
> [ERROR]
> throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
> Time elapsed: 0.154 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithoutRestartsHelper(JobMasterStopWithSavepointITCase.java:154)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(JobMasterStopWithSavepointITCase.java:138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dawidwys opened a new pull request #15829: [FLINK-22266][hotfix] Do not clean up jobs in AbstractTestBase if the MiniCluster is not running
dawidwys opened a new pull request #15829: URL: https://github.com/apache/flink/pull/15829 ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339088#comment-17339088
]
Dawid Wysakowicz commented on FLINK-22266:
--
General comment. I think there are two issues in here. One happening more
frequently occurs only with the adaptive scheduler and manifests with the
exception above. The other one is when we get the {{AskTimeoutException}}.
> Harden JobMasterStopWithSavepointITCase
> ---
>
> Key: FLINK-22266
> URL: https://issues.apache.org/jira/browse/FLINK-22266
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16451=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=3884
> {code:java}
> [ERROR]
> throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
> Time elapsed: 0.154 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithoutRestartsHelper(JobMasterStopWithSavepointITCase.java:154)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(JobMasterStopWithSavepointITCase.java:138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339086#comment-17339086
]
Dawid Wysakowicz edited comment on FLINK-22266 at 5/4/21, 3:33 PM:
---
There might actually be a problem in the AdaptiveScheduler. After investigating
some logs, I find the following stacktrace:
{code}
21:54:12,518 [flink-akka.actor.default-dispatcher-4] WARN
org.apache.flink.runtime.minicluster.MiniCluster [] - Error in
MiniCluster. Shutting the MiniCluster down.
java.lang.IllegalStateException: The vertex must be in CREATED or SCHEDULED
state to be deployed. Found state RUNNING
at
org.apache.flink.runtime.executiongraph.Execution.deploy(Execution.java:546)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.executiongraph.ExecutionVertex.deploy(ExecutionVertex.java:427)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing.deploySafely(Executing.java:139)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing.deploy(Executing.java:132)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing.(Executing.java:63)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing$Factory.getState(Executing.java:363)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing$Factory.getState(Executing.java:334)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.transitionToState(AdaptiveScheduler.java:1139)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.goToExecuting(AdaptiveScheduler.java:787)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.StopWithSavepoint.handleSavepointCompletion(StopWithSavepoint.java:106)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.StopWithSavepoint.lambda$null$0(StopWithSavepoint.java:89)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.runIfState(AdaptiveScheduler.java:1093)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.lambda$runIfState$26(AdaptiveScheduler.java:1108)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_282]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_282]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[scala-library-2.11.12.jar:?]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[scala-library-2.11.12.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[scala-library-2.11.12.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[scala-library-2.11.12.jar:?]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[jira] [Commented] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339086#comment-17339086
]
Dawid Wysakowicz commented on FLINK-22266:
--
There might actually be a problem in the AdaptiveScheduler. After investigating
some logs, I find the following stacktrace:
{code}
21:54:12,518 [flink-akka.actor.default-dispatcher-4] WARN
org.apache.flink.runtime.minicluster.MiniCluster [] - Error in
MiniCluster. Shutting the MiniCluster down.
java.lang.IllegalStateException: The vertex must be in CREATED or SCHEDULED
state to be deployed. Found state RUNNING
at
org.apache.flink.runtime.executiongraph.Execution.deploy(Execution.java:546)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.executiongraph.ExecutionVertex.deploy(ExecutionVertex.java:427)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing.deploySafely(Executing.java:139)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing.deploy(Executing.java:132)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing.(Executing.java:63)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing$Factory.getState(Executing.java:363)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.Executing$Factory.getState(Executing.java:334)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.transitionToState(AdaptiveScheduler.java:1139)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.goToExecuting(AdaptiveScheduler.java:787)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.StopWithSavepoint.handleSavepointCompletion(StopWithSavepoint.java:106)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.StopWithSavepoint.lambda$null$0(StopWithSavepoint.java:89)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.runIfState(AdaptiveScheduler.java:1093)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler.lambda$runIfState$26(AdaptiveScheduler.java:1108)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_282]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_282]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:440)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:208)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
~[flink-runtime_2.11-1.14-SNAPSHOT.jar:1.14-SNAPSHOT]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[scala-library-2.11.12.jar:?]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[scala-library-2.11.12.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[scala-library-2.11.12.jar:?]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[scala-library-2.11.12.jar:?]
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
[akka-actor_2.11-2.5.21.jar:2.5.21]
at
[GitHub] [flink] flinkbot edited a comment on pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
flinkbot edited a comment on pull request #15826: URL: https://github.com/apache/flink/pull/15826#issuecomment-831762997 ## CI report: * 6233c0163d44ca32bac1f7ec5dcfaec7d6aacb6a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17537) * ebdf6f053c6e8c2b07e40af8629a2a5954e37b10 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17545) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
flinkbot edited a comment on pull request #15826: URL: https://github.com/apache/flink/pull/15826#issuecomment-831762997 ## CI report: * 6233c0163d44ca32bac1f7ec5dcfaec7d6aacb6a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17537) * ebdf6f053c6e8c2b07e40af8629a2a5954e37b10 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21308) Cancel "sendAfter"
[
https://issues.apache.org/jira/browse/FLINK-21308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339042#comment-17339042
]
Igal Shilman commented on FLINK-21308:
--
Hi [~stephanpelikan]! Thanks for proposing this feature and taking a stab with
the implementation.
First of all, I do think that this can be useful, especially to long running
processes for months and years. So I'm eggar to move forward with this.
However I think that there are some tricky parts in this specific proposal.
1) I think it is important to reflect that undoing a send is a best effort
operation, and might actually be too late to undo a message.
2) I didn't understand why sendAfter now receives a Function<> ?
3) compatibility with previous snapshot
I don't think this would work when restoring from a savepoint:
[https://github.com/apache/flink-statefun/compare/release-2.2.1...stephanpelikan:feature/FLINK-21308_unsendAfter?expand=1#diff-170839ae52efa0bd357394ca6d8e3e3dd6ed4c0ae20ed4d548a2f50543637ae6R232]
cc: [~tzulitai]
4) This change needs to be also reflected in the remote SDK.
Overall, I think that overall, it would be better to proceed as follows:
1) Lets agree on a reasonable API for unending delayed messages and be very
explicit that this is a best effort thing.
2) Lets think what would be a good representation for a delated message id?
(perhaps a pair of : , maybe encoded as a string like
you proposed)
3) We need to be mindful about backwards compatibility here. And this is the
reason I'd suggest adding this ID into the Message class itself (should'nt be
user visible, only runtime)
And, under the assumption that a single function instance will have a
reasonable amount of timers, I'd suggest still remaining with a List state, and
removing
the desired message by a linear iteration.
What do you think [~stephanpelikan] and [~tzulitai]?
> Cancel "sendAfter"
> --
>
> Key: FLINK-21308
> URL: https://issues.apache.org/jira/browse/FLINK-21308
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
>Reporter: Stephan Pelikan
>Priority: Minor
> Labels: auto-deprioritized-major
>
> As a user I want to cancel delayed sent messages not needed any more to keep
> state clean.
> Use case:
> {quote}My use-case is processing business events of customers. Those events
> are triggered by ourself or by the customer depending of what's the current
> state of the ongoing customer's business use-case. We need to monitor
> delayed/missing business events which belong to previous events. For example:
> the customer has to confirm something we did. Depending on what it is the
> confirmation has to be within hours, days or even months. If there is a delay
> we need to know. But if the customer confirms in time we want to cleanup to
> keep the state small.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22557) Japicmp fails on 1.12 branch
[
https://issues.apache.org/jira/browse/FLINK-22557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-22557:
--
Fix Version/s: (was: 1.12.3)
1.12.4
> Japicmp fails on 1.12 branch
>
>
> Key: FLINK-22557
> URL: https://issues.apache.org/jira/browse/FLINK-22557
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.12.3
>Reporter: Dawid Wysakowicz
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.12.4
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17529=logs=ed6509f5-1153-558c-557a-5ee0afbcdf24=241b1e5e-1a8e-5e6a-469a-a9b8cad87065
> {code}
> 2021-05-03T21:26:13.4153009Z [ERROR] Failed to execute goal
> com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on
> project flink-scala_2.12: Breaking the build because there is at least one
> incompatibility:
>
[jira] [Assigned] (FLINK-22557) Japicmp fails on 1.12 branch
[
https://issues.apache.org/jira/browse/FLINK-22557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann reassigned FLINK-22557:
-
Assignee: Chesnay Schepler
> Japicmp fails on 1.12 branch
>
>
> Key: FLINK-22557
> URL: https://issues.apache.org/jira/browse/FLINK-22557
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.12.3
>Reporter: Dawid Wysakowicz
>Assignee: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.12.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17529=logs=ed6509f5-1153-558c-557a-5ee0afbcdf24=241b1e5e-1a8e-5e6a-469a-a9b8cad87065
> {code}
> 2021-05-03T21:26:13.4153009Z [ERROR] Failed to execute goal
> com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on
> project flink-scala_2.12: Breaking the build because there is at least one
> incompatibility:
>
[GitHub] [flink] flinkbot edited a comment on pull request #15728: [FLINK-22379][runtime] CheckpointCoordinator checks the state of all …
flinkbot edited a comment on pull request #15728: URL: https://github.com/apache/flink/pull/15728#issuecomment-824960796 ## CI report: * 057695465c326acc5c40576cf35ee9f0283d6cac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17524) * 87072bdee1da392386c3bd26b740f980ffee4017 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17544) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
flinkbot edited a comment on pull request #15826: URL: https://github.com/apache/flink/pull/15826#issuecomment-831762997 ## CI report: * 6233c0163d44ca32bac1f7ec5dcfaec7d6aacb6a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17537) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15728: [FLINK-22379][runtime] CheckpointCoordinator checks the state of all …
flinkbot edited a comment on pull request #15728: URL: https://github.com/apache/flink/pull/15728#issuecomment-824960796 ## CI report: * 057695465c326acc5c40576cf35ee9f0283d6cac Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17524) * 87072bdee1da392386c3bd26b740f980ffee4017 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22266) Harden JobMasterStopWithSavepointITCase
[
https://issues.apache.org/jira/browse/FLINK-22266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339002#comment-17339002
]
Dawid Wysakowicz commented on FLINK-22266:
--
Yes, I will, sorry for the delay.
> Harden JobMasterStopWithSavepointITCase
> ---
>
> Key: FLINK-22266
> URL: https://issues.apache.org/jira/browse/FLINK-22266
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Dawid Wysakowicz
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16451=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=f508e270-48d6-5f1e-3138-42a17e0714f0=3884
> {code:java}
> [ERROR]
> throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase)
> Time elapsed: 0.154 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithoutRestartsHelper(JobMasterStopWithSavepointITCase.java:154)
> at
> org.apache.flink.runtime.jobmaster.JobMasterStopWithSavepointITCase.throwingExceptionOnCallbackWithNoRestartsShouldFailTheTerminate(JobMasterStopWithSavepointITCase.java:138)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] maver1ck commented on pull request #15674: [FLINK-22311] [connector/jdbc] Validate maxRetries for XA Sink
maver1ck commented on pull request #15674: URL: https://github.com/apache/flink/pull/15674#issuecomment-831940983 @curcur Should I merge your commit to this branch ? Sorry I was absent last week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-22548) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to IllegalReferenceCountException
[
https://issues.apache.org/jira/browse/FLINK-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-22548.
--
Fix Version/s: (was: 1.13.1)
Resolution: Fixed
merged commit 1554292 into apache:master
I think this issue can not occur on the release-1.13 branch, as the extra
logging has never been merged there.
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to
> IllegalReferenceCountException
>
>
> Key: FLINK-22548
> URL: https://issues.apache.org/jira/browse/FLINK-22548
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17499=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9615
> {code:java}
> 2021-05-02T23:18:10.5449890Z May 02 23:18:10 [ERROR]
> shouldRescaleUnalignedCheckpoint[upscale pipeline from 1 to
> 2](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 2.645 s <<< ERROR!
> 2021-05-02T23:18:10.5451234Z May 02 23:18:10
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-05-02T23:18:10.5452049Z May 02 23:18:10 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-05-02T23:18:10.5453118Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:167)
> 2021-05-02T23:18:10.5454315Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
> 2021-05-02T23:18:10.5455228Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-02T23:18:10.5499169Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-02T23:18:10.5500258Z May 02 23:18:10 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-02T23:18:10.5501465Z May 02 23:18:10 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-02T23:18:10.5502547Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-02T23:18:10.5503965Z May 02 23:18:10 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-02T23:18:10.5504867Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-02T23:18:10.5505729Z May 02 23:18:10 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-02T23:18:10.5506527Z May 02 23:18:10 at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-05-02T23:18:10.5507319Z May 02 23:18:10 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-02T23:18:10.5508119Z May 02 23:18:10 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-02T23:18:10.5508896Z May 02 23:18:10 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-02T23:18:10.5509592Z May 02 23:18:10 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-02T23:18:10.5510318Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-02T23:18:10.5517041Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-02T23:18:10.5518247Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-05-02T23:18:10.5520181Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-05-02T23:18:10.5520928Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-05-02T23:18:10.5521660Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-05-02T23:18:10.5522323Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-05-02T23:18:10.5523524Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-05-02T23:18:10.5524309Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-05-02T23:18:10.5524934Z May 02 23:18:10 at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-05-02T23:18:10.5525581Z May 02 23:18:10 at
>
[jira] [Updated] (FLINK-22548) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to IllegalReferenceCountException
[
https://issues.apache.org/jira/browse/FLINK-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-22548:
---
Affects Version/s: (was: 1.13.0)
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to
> IllegalReferenceCountException
>
>
> Key: FLINK-22548
> URL: https://issues.apache.org/jira/browse/FLINK-22548
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0
>Reporter: Guowei Ma
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17499=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9615
> {code:java}
> 2021-05-02T23:18:10.5449890Z May 02 23:18:10 [ERROR]
> shouldRescaleUnalignedCheckpoint[upscale pipeline from 1 to
> 2](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 2.645 s <<< ERROR!
> 2021-05-02T23:18:10.5451234Z May 02 23:18:10
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-05-02T23:18:10.5452049Z May 02 23:18:10 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-05-02T23:18:10.5453118Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:167)
> 2021-05-02T23:18:10.5454315Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
> 2021-05-02T23:18:10.5455228Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-02T23:18:10.5499169Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-02T23:18:10.5500258Z May 02 23:18:10 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-02T23:18:10.5501465Z May 02 23:18:10 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-02T23:18:10.5502547Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-02T23:18:10.5503965Z May 02 23:18:10 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-02T23:18:10.5504867Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-02T23:18:10.5505729Z May 02 23:18:10 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-02T23:18:10.5506527Z May 02 23:18:10 at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-05-02T23:18:10.5507319Z May 02 23:18:10 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-02T23:18:10.5508119Z May 02 23:18:10 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-02T23:18:10.5508896Z May 02 23:18:10 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-02T23:18:10.5509592Z May 02 23:18:10 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-02T23:18:10.5510318Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-02T23:18:10.5517041Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-02T23:18:10.5518247Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-05-02T23:18:10.5520181Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-05-02T23:18:10.5520928Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-05-02T23:18:10.5521660Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-05-02T23:18:10.5522323Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-05-02T23:18:10.5523524Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-05-02T23:18:10.5524309Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-05-02T23:18:10.5524934Z May 02 23:18:10 at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-05-02T23:18:10.5525581Z May 02 23:18:10 at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2021-05-02T23:18:10.5526293Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>
[jira] [Updated] (FLINK-22232) Improve test coverage for network stack
[ https://issues.apache.org/jira/browse/FLINK-22232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-22232: --- Fix Version/s: (was: 1.13.1) 1.14.0 > Improve test coverage for network stack > --- > > Key: FLINK-22232 > URL: https://issues.apache.org/jira/browse/FLINK-22232 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Network, Tests >Reporter: Roman Khachatryan >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > This is a clone of FLINK-20103 to improve test coverage in the shorter term > and target specifically network stack. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15827: [FLINK-22555][build][python] Exclude leftover jboss files
flinkbot edited a comment on pull request #15827: URL: https://github.com/apache/flink/pull/15827#issuecomment-831839655 ## CI report: * 054af9b357b26f41fd879384b1a0fe3b35cc Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17540) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-21131) Show alignment timeout in checkpoint configuration (web UI)
[ https://issues.apache.org/jira/browse/FLINK-21131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roman Khachatryan resolved FLINK-21131. --- Fix Version/s: 1.14.0 Resolution: Fixed Merged into master as 53cf103cb431f43d46bdc7a82174de08f2422b8c. > Show alignment timeout in checkpoint configuration (web UI) > --- > > Key: FLINK-21131 > URL: https://issues.apache.org/jira/browse/FLINK-21131 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / Web Frontend >Affects Versions: 1.13.0 >Reporter: Roman Khachatryan >Assignee: Anton Kalashnikov >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: image-2021-01-25-13-56-30-832.png > > > !image-2021-01-25-13-56-30-832.png|width=825,height=426! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] pnowojski merged pull request #15828: [FLINK-22548][network] Remove illegal unsynchronized access to PipelinedSubpartition#buffers
pnowojski merged pull request #15828: URL: https://github.com/apache/flink/pull/15828 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan merged pull request #15772: [FLINK-21131][webui] Alignment timeout displayed in checkpoint config…
rkhachatryan merged pull request #15772: URL: https://github.com/apache/flink/pull/15772 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dannycranmer commented on a change in pull request #15826: [FLINK-17170][kinesis] Fix deadlock during stop-with-savepoint.
dannycranmer commented on a change in pull request #15826:
URL: https://github.com/apache/flink/pull/15826#discussion_r625742538
##
File path:
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisITCase.java
##
@@ -0,0 +1,191 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis;
+
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.common.serialization.SimpleStringSchema;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.runtime.client.JobStatusMessage;
+import org.apache.flink.runtime.testutils.CommonTestUtils;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.streaming.connectors.kinesis.config.AWSConfigConstants;
+import
org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.InitialPosition;
+import org.apache.flink.test.util.MiniClusterWithClientResource;
+import org.apache.flink.util.function.RunnableWithException;
+
+import com.amazonaws.services.kinesis.model.PutRecordsRequest;
+import com.amazonaws.services.kinesis.model.PutRecordsRequestEntry;
+import org.junit.After;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.testcontainers.utility.DockerImageName;
+
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.ForkJoinTask;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+import static
org.apache.flink.streaming.connectors.kinesis.config.ConsumerConfigConstants.STREAM_INITIAL_POSITION;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.hasSize;
+import static org.hamcrest.Matchers.lessThanOrEqualTo;
+import static org.junit.Assert.assertThat;
+
+/** IT cases for using Kinesis consumer/producer based on Kinesalite. */
+public class FlinkKinesisITCase {
Review comment:
Existing IT for Kinesis was added to the e2e module, inline with the
Kafka variant:
-
https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-streaming-kinesis-test/src/test/java/org/apache/flink/streaming/kinesis/test/KinesisTableApiITCase.java
I agree they are best kept within this module, however we should make them
consistent. We should at least raise a follow up Jira to tidy up
##
File path:
flink-connectors/flink-connector-kinesis/src/test/java/org/apache/flink/streaming/connectors/kinesis/KinesaliteContainer.java
##
@@ -0,0 +1,91 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis;
+
+import com.amazonaws.auth.AWSStaticCredentialsProvider;
+import com.amazonaws.auth.BasicAWSCredentials;
+import com.amazonaws.client.builder.AwsClientBuilder;
+import com.amazonaws.services.kinesis.AmazonKinesis;
+import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
+import org.rnorth.ducttape.unreliables.Unreliables;
+import org.testcontainers.containers.GenericContainer;
+import org.testcontainers.containers.wait.strategy.AbstractWaitStrategy;
+import
[GitHub] [flink] flinkbot edited a comment on pull request #15828: [FLINK-22548][network] Remove illegal unsynchronized access to PipelinedSubpartition#buffers
flinkbot edited a comment on pull request #15828: URL: https://github.com/apache/flink/pull/15828#issuecomment-831863456 ## CI report: * 0f517580ee7a2f8062913a8be96f01c95180c390 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17541) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol merged pull request #15738: modify annotation
zentol merged pull request #15738: URL: https://github.com/apache/flink/pull/15738 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15828: [FLINK-22548][network] Remove illegal unsynchronized access to PipelinedSubpartition#buffers
flinkbot commented on pull request #15828: URL: https://github.com/apache/flink/pull/15828#issuecomment-831863456 ## CI report: * 0f517580ee7a2f8062913a8be96f01c95180c390 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #15827: [FLINK-22555][build][python] Exclude leftover jboss files
zentol commented on a change in pull request #15827: URL: https://github.com/apache/flink/pull/15827#discussion_r625691675 ## File path: pom.xml ## @@ -1748,6 +1748,7 @@ under the License. log4j2.properties log4j-test.properties log4j2-test.properties + META-INF/maven/** Review comment: TODO: double-check that this also works properly for multi-release jars (e.g., flink-dist) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15827: [FLINK-22555][build][python] Exclude leftover jboss files
flinkbot edited a comment on pull request #15827: URL: https://github.com/apache/flink/pull/15827#issuecomment-831839655 ## CI report: * 054af9b357b26f41fd879384b1a0fe3b35cc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=17540) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22173) UnalignedCheckpointRescaleITCase fails on azure
[
https://issues.apache.org/jira/browse/FLINK-22173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338905#comment-17338905
]
Piotr Nowojski commented on FLINK-22173:
[~roman_khachatryan], your issue is unrelated to this one and it's already
reported as FLINK-22548
> UnalignedCheckpointRescaleITCase fails on azure
> ---
>
> Key: FLINK-22173
> URL: https://issues.apache.org/jira/browse/FLINK-22173
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Assignee: Arvid Heise
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16232=logs=d8d26c26-7ec2-5ed2-772e-7a1a1eb8317c=be5fb08e-1ad7-563c-4f1a-a97ad4ce4865=9628
> {code}
> 2021-04-08T23:25:56.3131361Z [ERROR] Tests run: 31, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 839.623 s <<< FAILURE! - in
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase
> 2021-04-08T23:25:56.3132784Z [ERROR] shouldRescaleUnalignedCheckpoint[no
> scale union from 7 to
> 7](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 607.467 s <<< ERROR!
> 2021-04-08T23:25:56.3133586Z
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-04-08T23:25:56.3134070Z at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-04-08T23:25:56.3134643Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:168)
> 2021-04-08T23:25:56.3135577Z at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
> 2021-04-08T23:25:56.3138843Z at
> sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
> 2021-04-08T23:25:56.3139402Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-04-08T23:25:56.3139880Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-04-08T23:25:56.3140328Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-04-08T23:25:56.3140844Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-04-08T23:25:56.3141768Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-04-08T23:25:56.3142272Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-04-08T23:25:56.3142706Z at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-04-08T23:25:56.3143142Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-08T23:25:56.3143608Z at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-04-08T23:25:56.3144039Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-04-08T23:25:56.3144434Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-04-08T23:25:56.3145027Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-04-08T23:25:56.3145484Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-04-08T23:25:56.3145981Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-04-08T23:25:56.3146421Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-04-08T23:25:56.3146843Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-04-08T23:25:56.3147274Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-04-08T23:25:56.3147692Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-04-08T23:25:56.3148116Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-04-08T23:25:56.3148543Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-04-08T23:25:56.3148930Z at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-04-08T23:25:56.3149298Z at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2021-04-08T23:25:56.3149663Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-04-08T23:25:56.3150075Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-04-08T23:25:56.3150488Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-04-08T23:25:56.3151148Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-04-08T23:25:56.3151691Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-04-08T23:25:56.3152115Z at
>
[GitHub] [flink] flinkbot commented on pull request #15828: [FLINK-22548][network] Remove illegal unsynchronized access to PipelinedSubpartition#buffers
flinkbot commented on pull request #15828: URL: https://github.com/apache/flink/pull/15828#issuecomment-831846667 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0f517580ee7a2f8062913a8be96f01c95180c390 (Tue May 04 10:41:16 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22548) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to IllegalReferenceCountException
[
https://issues.apache.org/jira/browse/FLINK-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-22548:
---
Labels: pull-request-available test-stability (was: test-stability)
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to
> IllegalReferenceCountException
>
>
> Key: FLINK-22548
> URL: https://issues.apache.org/jira/browse/FLINK-22548
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.14.0, 1.13.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17499=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9615
> {code:java}
> 2021-05-02T23:18:10.5449890Z May 02 23:18:10 [ERROR]
> shouldRescaleUnalignedCheckpoint[upscale pipeline from 1 to
> 2](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 2.645 s <<< ERROR!
> 2021-05-02T23:18:10.5451234Z May 02 23:18:10
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-05-02T23:18:10.5452049Z May 02 23:18:10 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-05-02T23:18:10.5453118Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:167)
> 2021-05-02T23:18:10.5454315Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
> 2021-05-02T23:18:10.5455228Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-02T23:18:10.5499169Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-02T23:18:10.5500258Z May 02 23:18:10 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-02T23:18:10.5501465Z May 02 23:18:10 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-02T23:18:10.5502547Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-02T23:18:10.5503965Z May 02 23:18:10 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-02T23:18:10.5504867Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-02T23:18:10.5505729Z May 02 23:18:10 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-02T23:18:10.5506527Z May 02 23:18:10 at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-05-02T23:18:10.5507319Z May 02 23:18:10 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-02T23:18:10.5508119Z May 02 23:18:10 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-02T23:18:10.5508896Z May 02 23:18:10 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-02T23:18:10.5509592Z May 02 23:18:10 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-02T23:18:10.5510318Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-02T23:18:10.5517041Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-02T23:18:10.5518247Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-05-02T23:18:10.5520181Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-05-02T23:18:10.5520928Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-05-02T23:18:10.5521660Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-05-02T23:18:10.5522323Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-05-02T23:18:10.5523524Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-05-02T23:18:10.5524309Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-05-02T23:18:10.5524934Z May 02 23:18:10 at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-05-02T23:18:10.5525581Z May 02 23:18:10 at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2021-05-02T23:18:10.5526293Z May 02 23:18:10 at
>
[GitHub] [flink] pnowojski opened a new pull request #15828: [FLINK-22548][network] Remove illegal unsynchronized access to PipelinedSubpartition#buffers
pnowojski opened a new pull request #15828: URL: https://github.com/apache/flink/pull/15828 After peeking last buffer, this last buffer could have been processed and recycled by the task thread, before NetworkActionsLogger.traceRecover would manage to check it's content causing IllegalReferenceCount exceptions. Further more, this log seemed excessive and was accidentally added after debugging session, so it's ok to remove it. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22548) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to IllegalReferenceCountException
[
https://issues.apache.org/jira/browse/FLINK-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338903#comment-17338903
]
Piotr Nowojski commented on FLINK-22548:
This is just a TRACE logging issue. There was indeed a potential illegal access
to buffer when printing some TRACE level logs, so it shouldn't affect any
users. And it wouldn't be causing any data corruptions even user enabled TRACE
logs. (fix is also trivial an will be merged soon)
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to
> IllegalReferenceCountException
>
>
> Key: FLINK-22548
> URL: https://issues.apache.org/jira/browse/FLINK-22548
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0, 1.13.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17499=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9615
> {code:java}
> 2021-05-02T23:18:10.5449890Z May 02 23:18:10 [ERROR]
> shouldRescaleUnalignedCheckpoint[upscale pipeline from 1 to
> 2](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 2.645 s <<< ERROR!
> 2021-05-02T23:18:10.5451234Z May 02 23:18:10
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-05-02T23:18:10.5452049Z May 02 23:18:10 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-05-02T23:18:10.5453118Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:167)
> 2021-05-02T23:18:10.5454315Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
> 2021-05-02T23:18:10.5455228Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-02T23:18:10.5499169Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-02T23:18:10.5500258Z May 02 23:18:10 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-02T23:18:10.5501465Z May 02 23:18:10 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-02T23:18:10.5502547Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-02T23:18:10.5503965Z May 02 23:18:10 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-02T23:18:10.5504867Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-02T23:18:10.5505729Z May 02 23:18:10 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-02T23:18:10.5506527Z May 02 23:18:10 at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-05-02T23:18:10.5507319Z May 02 23:18:10 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-02T23:18:10.5508119Z May 02 23:18:10 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-02T23:18:10.5508896Z May 02 23:18:10 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-02T23:18:10.5509592Z May 02 23:18:10 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-02T23:18:10.5510318Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-02T23:18:10.5517041Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-02T23:18:10.5518247Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-05-02T23:18:10.5520181Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-05-02T23:18:10.5520928Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-05-02T23:18:10.5521660Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-05-02T23:18:10.5522323Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-05-02T23:18:10.5523524Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-05-02T23:18:10.5524309Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-05-02T23:18:10.5524934Z May 02 23:18:10 at
>
[jira] [Commented] (FLINK-22555) LGPL-2.1 files in flink-python jars
[ https://issues.apache.org/jira/browse/FLINK-22555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338900#comment-17338900 ] Till Rohrmann commented on FLINK-22555: --- cc [~dianfu]. > LGPL-2.1 files in flink-python jars > --- > > Key: FLINK-22555 > URL: https://issues.apache.org/jira/browse/FLINK-22555 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.13.0, 1.12.3 >Reporter: Henri Yandell >Assignee: Chesnay Schepler >Priority: Blocker > Labels: pull-request-available > Fix For: 1.13.1, 1.12.4 > > > Looking at, for example, > [https://repo1.maven.org/maven2/org/apache/flink/flink-python_2.11/1.13.0/] > the jar file contains three LGPL-2.1 source files: > * > flink-python_2.11-1.13.0/META-INF/maven/org.jboss.modules/jboss-modules/pom.xml > * flink-python_2.11-1.13.0/schema/module-1_1.xsd > * flink-python_2.11-1.13.0/schema/module-1_0.xsd > There's nothing in the DEPENDENCIES or licenses directory on the topic. It > looks like Netty brings in the LGPL-2.1 JBoss dependency (depending on the > Apache-2.0 JBoss Marshalling, which depends on the LGPL-2.1 JBoss Modules). > The xsd files are also appear to be coming from JBoss Modules. Given Apache's > position on LGPL-2.1 dependency inclusion in ASF projects, this seems like an > issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-22548) UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to IllegalReferenceCountException
[
https://issues.apache.org/jira/browse/FLINK-22548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-22548:
--
Assignee: Piotr Nowojski
> UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint fail due to
> IllegalReferenceCountException
>
>
> Key: FLINK-22548
> URL: https://issues.apache.org/jira/browse/FLINK-22548
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.13.0, 1.14.0
>Reporter: Guowei Ma
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: test-stability
> Fix For: 1.14.0, 1.13.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17499=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9615
> {code:java}
> 2021-05-02T23:18:10.5449890Z May 02 23:18:10 [ERROR]
> shouldRescaleUnalignedCheckpoint[upscale pipeline from 1 to
> 2](org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase)
> Time elapsed: 2.645 s <<< ERROR!
> 2021-05-02T23:18:10.5451234Z May 02 23:18:10
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> 2021-05-02T23:18:10.5452049Z May 02 23:18:10 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> 2021-05-02T23:18:10.5453118Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointTestBase.execute(UnalignedCheckpointTestBase.java:167)
> 2021-05-02T23:18:10.5454315Z May 02 23:18:10 at
> org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase.shouldRescaleUnalignedCheckpoint(UnalignedCheckpointRescaleITCase.java:368)
> 2021-05-02T23:18:10.5455228Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-05-02T23:18:10.5499169Z May 02 23:18:10 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-05-02T23:18:10.5500258Z May 02 23:18:10 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-05-02T23:18:10.5501465Z May 02 23:18:10 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-05-02T23:18:10.5502547Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-05-02T23:18:10.5503965Z May 02 23:18:10 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-05-02T23:18:10.5504867Z May 02 23:18:10 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-05-02T23:18:10.5505729Z May 02 23:18:10 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-05-02T23:18:10.5506527Z May 02 23:18:10 at
> org.junit.rules.Verifier$1.evaluate(Verifier.java:35)
> 2021-05-02T23:18:10.5507319Z May 02 23:18:10 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-05-02T23:18:10.5508119Z May 02 23:18:10 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-05-02T23:18:10.5508896Z May 02 23:18:10 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-05-02T23:18:10.5509592Z May 02 23:18:10 at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-05-02T23:18:10.5510318Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-05-02T23:18:10.5517041Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-05-02T23:18:10.5518247Z May 02 23:18:10 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-05-02T23:18:10.5520181Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-05-02T23:18:10.5520928Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-05-02T23:18:10.5521660Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-05-02T23:18:10.5522323Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-05-02T23:18:10.5523524Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-05-02T23:18:10.5524309Z May 02 23:18:10 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-05-02T23:18:10.5524934Z May 02 23:18:10 at
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2021-05-02T23:18:10.5525581Z May 02 23:18:10 at
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2021-05-02T23:18:10.5526293Z May 02 23:18:10 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-05-02T23:18:10.5526966Z
[GitHub] [flink] flinkbot commented on pull request #15827: [FLINK-22555][build][python] Exclude leftover jboss files
flinkbot commented on pull request #15827: URL: https://github.com/apache/flink/pull/15827#issuecomment-831839655 ## CI report: * 054af9b357b26f41fd879384b1a0fe3b35cc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22562) Translate recent updates to backpressure and checkpointing docs
Piotr Nowojski created FLINK-22562: -- Summary: Translate recent updates to backpressure and checkpointing docs Key: FLINK-22562 URL: https://issues.apache.org/jira/browse/FLINK-22562 Project: Flink Issue Type: Bug Components: chinese-translation Affects Versions: 1.14.0, 1.13.1 Reporter: Piotr Nowojski https://issues.apache.org/jira/browse/FLINK-22253 https://github.com/apache/flink/pull/15811 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22253) Update backpressure monitoring documentation
[ https://issues.apache.org/jira/browse/FLINK-22253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338897#comment-17338897 ] Piotr Nowojski edited comment on FLINK-22253 at 5/4/21, 10:24 AM: -- merged to master as 368b2643eb0 and baf27ef7266 merged to release-1.13 as 5bd486e0a8c and 9ff2602ab7e was (Author: pnowojski): merged to master as baf27ef7266 merged to release-1.13 as 9ff2602ab7e > Update backpressure monitoring documentation > > > Key: FLINK-22253 > URL: https://issues.apache.org/jira/browse/FLINK-22253 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / Network >Affects Versions: 1.13.0 >Reporter: Dawid Wysakowicz >Assignee: Piotr Nowojski >Priority: Critical > Labels: pull-request-available, stale-critical > Fix For: 1.14.0, 1.13.1 > > > We should update the page: > [https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/monitoring/back_pressure/] > with the new backpressure monitoring mechanism. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-22253) Update backpressure monitoring documentation
[ https://issues.apache.org/jira/browse/FLINK-22253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski closed FLINK-22253. -- Fix Version/s: 1.13.1 Resolution: Fixed merged to master as baf27ef7266 merged to release-1.13 as 9ff2602ab7e > Update backpressure monitoring documentation > > > Key: FLINK-22253 > URL: https://issues.apache.org/jira/browse/FLINK-22253 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / Network >Affects Versions: 1.13.0 >Reporter: Dawid Wysakowicz >Assignee: Piotr Nowojski >Priority: Critical > Labels: pull-request-available, stale-critical > Fix For: 1.14.0, 1.13.1 > > > We should update the page: > [https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/monitoring/back_pressure/] > with the new backpressure monitoring mechanism. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22535) Resource leak would happen if exception thrown during AbstractInvokable#restore of task life
[
https://issues.apache.org/jira/browse/FLINK-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338367#comment-17338367
]
Piotr Nowojski edited comment on FLINK-22535 at 5/4/21, 10:20 AM:
--
Merged to master 5926c4ac22b^..5926c4ac22b
Merged to release-1.13 as ae37cde62b3^..ae37cde62b3
was (Author: pnowojski):
Merged to master 5926c4ac22b^..5926c4ac22b
Merged to release-1.13 as 0077b6fb07c^..0077b6fb07c
> Resource leak would happen if exception thrown during
> AbstractInvokable#restore of task life
>
>
> Key: FLINK-22535
> URL: https://issues.apache.org/jira/browse/FLINK-22535
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.13.0
>Reporter: Yun Tang
>Assignee: Anton Kalashnikov
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.1
>
>
> FLINK-17012 introduced new initialization phase such as
> {{AbstractInvokable.restore}}, however, if
> [invokable.restore()|https://github.com/apache/flink/blob/79a521e08df550d96f97bb6915191d8496bb29ea/flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java#L754-L759]
> throws exception out, no more {{StreamTask#cleanUpInvoke}} would be called,
> leading to resource leak.
> We internally leveraged another way to use managed memory by registering
> specific operator identifier in memory manager, forgetting to call the stream
> task cleanup would let stream operator not be disposed and we have to face
> critical resource leak.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] pnowojski merged pull request #15811: [FLINK-22253][docs] Update back pressure monitoring docs with new WebUI changes
pnowojski merged pull request #15811: URL: https://github.com/apache/flink/pull/15811 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on pull request #15811: [FLINK-22253][docs] Update back pressure monitoring docs with new WebUI changes
pnowojski commented on pull request #15811: URL: https://github.com/apache/flink/pull/15811#issuecomment-831834103 Thanks a lot! I've included that last one suggestion and I'm going to merge it soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15827: [FLINK-22555][build][python] Exclude leftover jboss files
flinkbot commented on pull request #15827: URL: https://github.com/apache/flink/pull/15827#issuecomment-831829389 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 054af9b357b26f41fd879384b1a0fe3b35cc (Tue May 04 10:08:55 UTC 2021) **Warnings:** * **11 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #15789: [FLINK-21181][runtime] Wait for Invokable cancellation before releasing network resources
dawidwys commented on a change in pull request #15789:
URL: https://github.com/apache/flink/pull/15789#discussion_r625661050
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java
##
@@ -1574,6 +1574,12 @@ public void run() {
// we need do continue despite that
try {
invokable.cancel();
+// Wait for any active actions to complete (e.g. timers,
mailbox actions)
+// Before that, interrupt to notify them know about
cancellation
+if (invokable.shouldInterruptOnCancel()) {
+executer.interrupt();
+}
+invokable.getTerminationFuture().join();
Review comment:
I see. Ok, that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
