[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 8e1e9761a59b5a271c9196fbc206e7dae4cea79b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24409) * a6fea2c00747f94e41c4e40497be5d35479c00eb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24417) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 8e1e9761a59b5a271c9196fbc206e7dae4cea79b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24409) * a6fea2c00747f94e41c4e40497be5d35479c00eb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on pull request #4: URL: https://github.com/apache/flink-ml/pull/4#issuecomment-925506182 @zhipeng93 Sure. Can you explain the concern? And do you think these concern needs to be addressed in this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
akalash commented on a change in pull request #17331:
URL: https://github.com/apache/flink/pull/17331#discussion_r714462624
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##
@@ -53,6 +56,56 @@ public CheckpointFailureManager(int
tolerableCpFailureNumber, FailJobCallback fa
this.countedCheckpointIds = ConcurrentHashMap.newKeySet();
}
+/**
+ * Failures on JM:
+ *
+ *
+ * all checkpoints - go against failure counter.
+ * any savepoints - don’t do anything, manual action, the failover
will not help anyway.
+ *
+ *
+ * Failures on TM:
+ *
+ *
+ * all checkpoints - go against failure counter (failover might help
and we want to notify
+ * users).
+ * sync savepoints - we must always fail, otherwise we risk deadlock
when the job
+ * cancelation waiting for finishing savepoint which never happens.
+ * non sync savepoints - go against failure counter (failover might
help solve the
+ * problem).
+ *
+ *
+ * @param pendingCheckpoint the failed checkpoint if it was initialized
already.
+ * @param checkpointProperties the checkpoint properties in order to
determinate which handle
+ * strategy can be used.
+ * @param exception the checkpoint exception.
+ * @param executionAttemptID the execution attempt id, as a safe guard.
+ */
+public void handleCheckpointException(
+@Nullable PendingCheckpoint pendingCheckpoint,
+CheckpointProperties checkpointProperties,
+CheckpointException exception,
+@Nullable ExecutionAttemptID executionAttemptID) {
+if (isJMFailure(pendingCheckpoint, exception, executionAttemptID)) {
+handleJobLevelCheckpointException(
+checkpointProperties,
+exception,
+pendingCheckpoint == null
+? UNKNOWN_CHECKPOINT_ID
+: pendingCheckpoint.getCheckpointID());
+} else {
+handleTaskLevelCheckpointException(pendingCheckpoint, exception,
executionAttemptID);
+}
+}
+
+/** Check if the exception occurs on the job manager side or not. */
+private boolean isJMFailure(
+@Nullable PendingCheckpoint pendingCheckpoint,
+CheckpointException exception,
+@Nullable ExecutionAttemptID executionAttemptID) {
+return pendingCheckpoint == null || isJMThrowable(exception) ||
executionAttemptID == null;
Review comment:
I agree that CheckpointFailureReason#isPreFlight should be enough but it
was just extra protection.
I removed `pendingCheckpoint == null` because it indeed can rely on
isPreFlight method but `executionAttemptID == null` is more complicated so I
left it as you proposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
zhipeng93 commented on pull request #4: URL: https://github.com/apache/flink-ml/pull/4#issuecomment-925490704 > @zhipeng93 Thanks for the review :) > > Can we delay the review/code of the save/load and also related implementation of Pipeline/PipelineModel to a later PR? The code/test for these features are available in https://github.com/lindong28/flink-ml/tree/flink-ml if you want to take a look earlier to address the concern regarding e.g. lazy sink. @lindong28 Thanks :) I checked the source code in the above link. I am still curious and concerned about how save(String path) & load(String path) work in a distributed setting. There are also some other related issues like failover if we are not using `sink` in Flink. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24293) Tasks from the same job on a machine share user jar
[ https://issues.apache.org/jira/browse/FLINK-24293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418915#comment-17418915 ] huntercc commented on FLINK-24293: -- hi [~jark], I have added some meaningful content and reopened this issue last week, I would appreciate it if there is some feedback about the issue. > Tasks from the same job on a machine share user jar > > > Key: FLINK-24293 > URL: https://issues.apache.org/jira/browse/FLINK-24293 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: huntercc >Priority: Major > Labels: pull-request-available > Attachments: image-2021-09-15-20-43-11-758.png, > image-2021-09-15-20-43-17-304.png > > > In the current blob storage design, tasks executed by the same TaskExecutor > will share BLOBs storage dir and tasks executed by different TaskExecutor use > different dir. As a result, a TaskExecutor has to download user jar even if > there has been the same user jar downloaded by other TaskExecutors on the > machine. We believe that there is no need to download many copies of the same > user jar to the local, two main problems will by exposed: > # The NIC bandwidth of the distribution terminal may become a bottleneck > !image-2021-09-15-20-43-17-304.png|width=695,height=193! > As shown in the figure above, 24640 Mbps of the total 25000 Mbps NIC > bandwidth is used when we launched a flink job with 4000 TaskManagers, which > will cause a long deployment time and akka timeout exception. > # Take up more disk space > We expect to optimize the sharing mechanism of user jar by allowing tasks > from the same job on a machine to share blob storage dir, more specifically, > share the user jar in the dir. Only one task deployed to the machine will > download the user jar from BLOB server or distributed file storage, and the > subsequent tasks just use the localized user jar. In this way, the user jar > of one job only needs to be downloaded once on a machine. Here is a > comparison of job startup time before and after optimization. > ||num of TM||before optimization||after optimization|| > |1000|62s|37s| > |2000|104s|40s| > |3000|170s|43s| > |4000|211s|45s| > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 8e1e9761a59b5a271c9196fbc206e7dae4cea79b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24409) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24358) flink-avro-glue-schema-registry fails compiling with dependency convergence
[
https://issues.apache.org/jira/browse/FLINK-24358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418923#comment-17418923
]
Xintong Song commented on FLINK-24358:
--
It's wired this does not fail other stages.
cc [~chesnay], any insights?
> flink-avro-glue-schema-registry fails compiling with dependency convergence
> ---
>
> Key: FLINK-24358
> URL: https://issues.apache.org/jira/browse/FLINK-24358
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Major
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24405=logs=ed6509f5-1153-558c-557a-5ee0afbcdf24=241b1e5e-1a8e-5e6a-469a-a9b8cad87065=13506
> {code}
> [WARNING]
> Dependency convergence error for io.netty:netty-handler:4.1.63.Final paths to
> dependency are:
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-common:1.1.2
> +-software.amazon.awssdk:glue:2.16.92
> +-software.amazon.awssdk:netty-nio-client:2.16.92
> +-io.netty:netty-codec-http:4.1.63.Final
> +-io.netty:netty-handler:4.1.63.Final
> and
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-common:1.1.2
> +-software.amazon.awssdk:glue:2.16.92
> +-software.amazon.awssdk:netty-nio-client:2.16.92
> +-io.netty:netty-codec-http2:4.1.63.Final
> +-io.netty:netty-handler:4.1.63.Final
> and
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-common:1.1.2
> +-software.amazon.awssdk:glue:2.16.92
> +-software.amazon.awssdk:netty-nio-client:2.16.92
> +-io.netty:netty-handler:4.1.63.Final
> and
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-common:1.1.2
> +-software.amazon.awssdk:glue:2.16.92
> +-software.amazon.awssdk:netty-nio-client:2.16.92
> +-com.typesafe.netty:netty-reactive-streams-http:2.0.5
> +-com.typesafe.netty:netty-reactive-streams:2.0.5
> +-io.netty:netty-handler:4.1.52.Final
> [WARNING]
> Dependency convergence error for org.jetbrains.kotlin:kotlin-stdlib:1.3.50
> paths to dependency are:
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-serde:1.1.2
> +-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
>
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
> +-org.jetbrains.kotlin:kotlin-scripting-common:1.3.50
> +-org.jetbrains.kotlin:kotlin-reflect:1.3.50
> +-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
> and
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-serde:1.1.2
> +-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
>
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
> +-org.jetbrains.kotlin:kotlin-scripting-common:1.3.50
> +-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
> and
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-serde:1.1.2
> +-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
>
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
> +-org.jetbrains.kotlin:kotlin-scripting-jvm:1.3.50
> +-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
> and
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-serde:1.1.2
> +-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
>
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
> +-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
> and
> +-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
> +-software.amazon.glue:schema-registry-serde:1.1.2
> +-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
>
> +-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
> +-org.jetbrains.kotlinx:kotlinx-coroutines-core:1.1.1
> +-org.jetbrains.kotlin:kotlin-stdlib:1.3.20
> and
>
[jira] [Created] (FLINK-24358) flink-avro-glue-schema-registry fails compiling with dependency convergence
Xintong Song created FLINK-24358:
Summary: flink-avro-glue-schema-registry fails compiling with
dependency convergence
Key: FLINK-24358
URL: https://issues.apache.org/jira/browse/FLINK-24358
Project: Flink
Issue Type: Bug
Components: Build System
Affects Versions: 1.15.0
Reporter: Xintong Song
Fix For: 1.15.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24405=logs=ed6509f5-1153-558c-557a-5ee0afbcdf24=241b1e5e-1a8e-5e6a-469a-a9b8cad87065=13506
{code}
[WARNING]
Dependency convergence error for io.netty:netty-handler:4.1.63.Final paths to
dependency are:
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-common:1.1.2
+-software.amazon.awssdk:glue:2.16.92
+-software.amazon.awssdk:netty-nio-client:2.16.92
+-io.netty:netty-codec-http:4.1.63.Final
+-io.netty:netty-handler:4.1.63.Final
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-common:1.1.2
+-software.amazon.awssdk:glue:2.16.92
+-software.amazon.awssdk:netty-nio-client:2.16.92
+-io.netty:netty-codec-http2:4.1.63.Final
+-io.netty:netty-handler:4.1.63.Final
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-common:1.1.2
+-software.amazon.awssdk:glue:2.16.92
+-software.amazon.awssdk:netty-nio-client:2.16.92
+-io.netty:netty-handler:4.1.63.Final
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-common:1.1.2
+-software.amazon.awssdk:glue:2.16.92
+-software.amazon.awssdk:netty-nio-client:2.16.92
+-com.typesafe.netty:netty-reactive-streams-http:2.0.5
+-com.typesafe.netty:netty-reactive-streams:2.0.5
+-io.netty:netty-handler:4.1.52.Final
[WARNING]
Dependency convergence error for org.jetbrains.kotlin:kotlin-stdlib:1.3.50
paths to dependency are:
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-serde:1.1.2
+-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
+-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-common:1.3.50
+-org.jetbrains.kotlin:kotlin-reflect:1.3.50
+-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-serde:1.1.2
+-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
+-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-common:1.3.50
+-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-serde:1.1.2
+-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
+-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-jvm:1.3.50
+-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-serde:1.1.2
+-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
+-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-serde:1.1.2
+-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
+-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-scripting-compiler-impl-embeddable:1.3.50
+-org.jetbrains.kotlinx:kotlinx-coroutines-core:1.1.1
+-org.jetbrains.kotlin:kotlin-stdlib:1.3.20
and
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-serde:1.1.2
+-com.kjetland:mbknor-jackson-jsonschema_2.12:1.0.39
+-org.jetbrains.kotlin:kotlin-scripting-compiler-embeddable:1.3.50
+-org.jetbrains.kotlin:kotlin-stdlib:1.3.50
[WARNING]
Dependency convergence error for io.netty:netty-codec-http:4.1.63.Final paths
to dependency are:
+-org.apache.flink:flink-avro-glue-schema-registry_2.12:1.15-SNAPSHOT
+-software.amazon.glue:schema-registry-common:1.1.2
[GitHub] [flink] flinkbot edited a comment on pull request #17336: [BP-1.13][FLINK-24315][k8s] Throw exception in failing watching pods
flinkbot edited a comment on pull request #17336: URL: https://github.com/apache/flink/pull/17336#issuecomment-925471041 ## CI report: * 80706fc5e305518c2060dfdf6341f88fc6f61ff1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24415) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN * 765eb79602d0d359ad2267f97cc5785663016c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24398) Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24411) * 32155a2dee456995136f4b13d83f703dc7bacdfc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24412) * 68773afc90e3de41db347b97cf884a264a7a4b59 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24414) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24416) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
xuyangzhong commented on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-925474981 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17336: [BP-1.13][FLINK-24315][k8s] Throw exception in failing watching pods
flinkbot commented on pull request #17336: URL: https://github.com/apache/flink/pull/17336#issuecomment-925471041 ## CI report: * 80706fc5e305518c2060dfdf6341f88fc6f61ff1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN * 765eb79602d0d359ad2267f97cc5785663016c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24398) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24411) * 32155a2dee456995136f4b13d83f703dc7bacdfc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24412) * 68773afc90e3de41db347b97cf884a264a7a4b59 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714439455
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,102 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink-ml] lindong28 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714438174
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,102 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink-ml] lindong28 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
lindong28 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714438112
##
File path:
flink-ml-api/src/main/java/org/apache/flink/ml/api/core/PipelineModel.java
##
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.pipeline;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.ml.api.core.AlgoOperator;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.table.api.Table;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * A PipelineModel acts as a Model. It consists of an ordered list of stages,
each of which could be
+ * a Model, Transformer or AlgoOperator.
+ */
+@PublicEvolving
+public final class PipelineModel implements Model {
+private static final long serialVersionUID = 6184950154217411318L;
+private final List> stages;
+private final Params params = new Params();
+
+public PipelineModel(List> stages) {
+this.stages = stages;
+}
+
+/**
+ * Applies all stages in this PipelineModel on the input tables in order.
The output of one
+ * stage is used as the input of the next stage (if any). The output of
the last stage is
+ * returned as the result of this method.
+ *
+ * @param inputs a list of tables
+ * @return a list of tables
+ */
+@Override
+public Table[] transform(Table... inputs) {
+for (Stage stage : stages) {
+inputs = ((AlgoOperator) stage).transform(inputs);
+}
+return inputs;
+}
+
+@Override
+public void save(String path) throws IOException {
Review comment:
I have the code for save/load as well as related tests ready. These code
are not uploaded for review yet because I am waiting for FLIP-174 to be
reviewed and accepted.
The following statement is put in the PR description:
`This PR focuses on changing the Flink ML API. Some APIs such as save/load
needs to be implemented after we complete the design of the parameter
interface`.
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,102 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
[jira] [Closed] (FLINK-23739) PrintTableSink do not implement SupportsPartitioning interface
[
https://issues.apache.org/jira/browse/FLINK-23739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-23739.
Fix Version/s: 1.15.0
Resolution: Fixed
master: 038f78ffa19c7e8b3dbd3da60c0e5054d575a870
> PrintTableSink do not implement SupportsPartitioning interface
> --
>
> Key: FLINK-23739
> URL: https://issues.apache.org/jira/browse/FLINK-23739
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.12.4
>Reporter: Xianxun Ye
>Assignee: Xianxun Ye
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.15.0
>
>
>
> {code:java}
> //代码占位符
> tEnv.executeSql(
> "CREATE TABLE PrintTable (name STRING, score INT, da STRING, hr
> STRING)\n"
> + "PARTITIONED BY (da, hr)"
> + "WITH (\n"
> + " 'connector' = 'print'"
> + ")");
> tEnv.executeSql("INSERT INTO PrintTable SELECT 'n1' as name, 1 as score,
> '2021-08-12' as da, '11' as hr");
> {code}
> Now print records with a partitioned table is not supported.
> {code:java}
> //代码占位符
> Exception in thread "main" org.apache.flink.table.api.TableException: Table
> 'default_catalog.default_database.PrintTable' is a partitioned table, but the
> underlying DynamicTableSink doesn't implement the SupportsPartitioning
> interface.Exception in thread "main"
> org.apache.flink.table.api.TableException: Table
> 'default_catalog.default_database.PrintTable' is a partitioned table, but the
> underlying DynamicTableSink doesn't implement the SupportsPartitioning
> interface. at
> org.apache.flink.table.planner.sinks.DynamicSinkUtils.validatePartitioning(DynamicSinkUtils.java:345)
> at
> org.apache.flink.table.planner.sinks.DynamicSinkUtils.prepareDynamicSink(DynamicSinkUtils.java:260)
> at
> org.apache.flink.table.planner.sinks.DynamicSinkUtils.toRel(DynamicSinkUtils.java:87)
> {code}
> `org.apache.flink.table.factories.PrintTableSinkFactory$PrintSink` and
> `org.apache.flink.table.factories.BlackHoleTableSinkFactory$BlackHoleSink`
> shoud implement `SupportsPartitioning` interface.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi merged pull request #16792: [FLINK-23739][table]BlackHoleSink & PrintSink implement SupportsParti…
JingsongLi merged pull request #16792: URL: https://github.com/apache/flink/pull/16792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24338) Provide an interface to validate flinksql
[
https://issues.apache.org/jira/browse/FLINK-24338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418908#comment-17418908
]
Jark Wu commented on FLINK-24338:
-
If you want to just validate SQL sytnax, maybe you can try to use this method
{{org.apache.flink.table.api.internal.TableEnvironmentImpl.getPlanner().getParser().parse(statement)}}.
> Provide an interface to validate flinksql
> -
>
> Key: FLINK-24338
> URL: https://issues.apache.org/jira/browse/FLINK-24338
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Spongebob
>Priority: Minor
>
> It would be great if there is an interface that can validate flinksql.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-24353) Bash scripts do not respect dynamic configurations when calculating memory sizes
[
https://issues.apache.org/jira/browse/FLINK-24353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418471#comment-17418471
]
Xintong Song edited comment on FLINK-24353 at 9/23/21, 2:26 AM:
I think both `{{-Dkey=value}}` and `{{-D key=value}}` are expected to work.
I tried this on {{taskmanager.numberOfTaskSlots}} and both ways work.
However, for memory options, -only `{{-Dkey=value}}` works but not `{{-D
key=value}}`- neither way works.
was (Author: xintongsong):
I think both `{{-Dkey=value}}` and `{{-D key=value}}` are expected to work.
I tried this on {{taskmanager.numberOfTaskSlots}} and both ways work.
However, for memory options, only `{{-Dkey=value}}` works but not `{{-D
key=value}}`.
> Bash scripts do not respect dynamic configurations when calculating memory
> sizes
>
>
> Key: FLINK-24353
> URL: https://issues.apache.org/jira/browse/FLINK-24353
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Scripts
>Affects Versions: 1.14.0, 1.13.2, 1.15.0
>Reporter: Xintong Song
>Assignee: Xintong Song
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.13.3, 1.15.0
>
>
> Dynamic configurations (the '-D' arguments) are lost due to changes in
> FLINK-21128.
> Consequently, dynamic configurations like the following commands will not
> take effect.
> {code}
> ./bin/taskmanager.sh start -D taskmanager.memory.task.off-heap.size=128m
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN * 765eb79602d0d359ad2267f97cc5785663016c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24398) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24411) * 32155a2dee456995136f4b13d83f703dc7bacdfc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] libenchao commented on pull request #13081: [FLINK-18590][json] Support json array explode to multi messages
libenchao commented on pull request #13081: URL: https://github.com/apache/flink/pull/13081#issuecomment-925459467 @slinkydeveloper Thanks for your interest, I'll rebase this pr and let's move forward. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17336: [BP-1.13][FLINK-24315][k8s] Throw exception in failing watching pods
flinkbot commented on pull request #17336: URL: https://github.com/apache/flink/pull/17336#issuecomment-925459351 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 80706fc5e305518c2060dfdf6341f88fc6f61ff1 (Thu Sep 23 02:15:44 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] KarmaGYZ opened a new pull request #17336: [BP-1.13][FLINK-24315][k8s] Throw exception in failing watching pods
KarmaGYZ opened a new pull request #17336: URL: https://github.com/apache/flink/pull/17336 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
zhipeng93 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714414798
##
File path:
flink-ml-api/src/main/java/org/apache/flink/ml/api/core/PipelineModel.java
##
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.api.pipeline;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.ml.api.core.AlgoOperator;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.api.core.Stage;
+import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.table.api.Table;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.List;
+
+/**
+ * A PipelineModel acts as a Model. It consists of an ordered list of stages,
each of which could be
+ * a Model, Transformer or AlgoOperator.
+ */
+@PublicEvolving
+public final class PipelineModel implements Model {
+private static final long serialVersionUID = 6184950154217411318L;
+private final List> stages;
+private final Params params = new Params();
+
+public PipelineModel(List> stages) {
+this.stages = stages;
+}
+
+/**
+ * Applies all stages in this PipelineModel on the input tables in order.
The output of one
+ * stage is used as the input of the next stage (if any). The output of
the last stage is
+ * returned as the result of this method.
+ *
+ * @param inputs a list of tables
+ * @return a list of tables
+ */
+@Override
+public Table[] transform(Table... inputs) {
+for (Stage stage : stages) {
+inputs = ((AlgoOperator) stage).transform(inputs);
+}
+return inputs;
+}
+
+@Override
+public void save(String path) throws IOException {
Review comment:
can we support load() and save() here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
zhipeng93 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714414999
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,102 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN * 765eb79602d0d359ad2267f97cc5785663016c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24398) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24411) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17305: [hotfix][flink-avro] Use local actual schema variable
flinkbot edited a comment on pull request #17305: URL: https://github.com/apache/flink/pull/17305#issuecomment-921331062 ## CI report: * dfc9f4122197a6ebea6041cf39cdd2681c26bcdd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24310) * e962467ac5dfff2a6c6ed1f9b771435bc0a7a507 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24410) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
xuyangzhong commented on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-925447627 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
RocMarshal commented on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-925446816 Hi, @MartijnVisser @Airblader , I made some changes based on your suggestions. Would you like to review it for me? Thank you so much for your attention. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
zhipeng93 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714413715
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,102 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
zhipeng93 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714414312
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,102 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink-ml] zhipeng93 commented on a change in pull request #4: [FLINK-22915][FLIP-173] Update Flink ML API to support AlgoOperator with multiple input tables and multiple output tables
zhipeng93 commented on a change in pull request #4:
URL: https://github.com/apache/flink-ml/pull/4#discussion_r714413715
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/api/core/Pipeline.java
##
@@ -19,241 +19,102 @@
package org.apache.flink.ml.api.core;
import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.ml.api.misc.param.Params;
+import org.apache.flink.ml.api.pipeline.PipelineModel;
import org.apache.flink.table.api.Table;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.util.InstantiationUtil;
-
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonProcessingException;
-import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
-import java.util.HashMap;
import java.util.List;
-import java.util.Map;
/**
- * A pipeline is a linear workflow which chains {@link Estimator}s and {@link
Transformer}s to
- * execute an algorithm.
- *
- * A pipeline itself can either act as an Estimator or a Transformer,
depending on the stages it
- * includes. More specifically:
- *
- *
- * If a Pipeline has an {@link Estimator}, one needs to call {@link
- * Pipeline#fit(TableEnvironment, Table)} before use the pipeline as a
{@link Transformer} .
- * In this case the Pipeline is an {@link Estimator} and can produce a
Pipeline as a {@link
- * Model}.
- * If a Pipeline has no {@link Estimator}, it is a {@link Transformer}
and can be applied to a
- * Table directly. In this case, {@link Pipeline#fit(TableEnvironment,
Table)} will simply
- * return the pipeline itself.
- *
- *
- * In addition, a pipeline can also be used as a {@link PipelineStage} in
another pipeline, just
- * like an ordinary {@link Estimator} or {@link Transformer} as describe above.
+ * A Pipeline acts as an Estimator. It consists of an ordered list of stages,
each of which could be
+ * an Estimator, Model, Transformer or AlgoOperator.
*/
@PublicEvolving
-public final class Pipeline
-implements Estimator, Transformer,
Model {
-private static final long serialVersionUID = 1L;
-private final List stages = new ArrayList<>();
+public final class Pipeline implements Estimator {
+private static final long serialVersionUID = 6384850154817512318L;
+private final List> stages;
private final Params params = new Params();
-private int lastEstimatorIndex = -1;
-
-public Pipeline() {}
-
-public Pipeline(String pipelineJson) {
-this.loadJson(pipelineJson);
-}
-
-public Pipeline(List stages) {
-for (PipelineStage s : stages) {
-appendStage(s);
-}
-}
-
-// is the stage a simple Estimator or pipeline with Estimator
-private static boolean isStageNeedFit(PipelineStage stage) {
-return (stage instanceof Pipeline && ((Pipeline) stage).needFit())
-|| (!(stage instanceof Pipeline) && stage instanceof
Estimator);
+public Pipeline(List> stages) {
+this.stages = stages;
}
/**
- * Appends a PipelineStage to the tail of this pipeline. Pipeline is
editable only via this
- * method. The PipelineStage must be Estimator, Transformer, Model or
Pipeline.
+ * Trains the pipeline to fit on the given tables.
*
- * @param stage the stage to be appended
- */
-public Pipeline appendStage(PipelineStage stage) {
-if (isStageNeedFit(stage)) {
-lastEstimatorIndex = stages.size();
-} else if (!(stage instanceof Transformer)) {
-throw new RuntimeException(
-"All PipelineStages should be Estimator or Transformer,
got:"
-+ stage.getClass().getSimpleName());
-}
-stages.add(stage);
-return this;
-}
-
-/**
- * Returns a list of all stages in this pipeline in order, the list is
immutable.
- *
- * @return an immutable list of all stages in this pipeline in order.
- */
-public List getStages() {
-return Collections.unmodifiableList(stages);
-}
-
-/**
- * Check whether the pipeline acts as an {@link Estimator} or not. When
the return value is
- * true, that means this pipeline contains an {@link Estimator} and thus
users must invoke
- * {@link #fit(TableEnvironment, Table)} before they can use this pipeline
as a {@link
- * Transformer}. Otherwise, the pipeline can be used as a {@link
Transformer} directly.
- *
- * @return {@code true} if this pipeline has an Estimator, {@code false}
otherwise
- */
-public boolean needFit() {
-return this.getIndexOfLastEstimator() >= 0;
-}
-
-public Params getParams() {
-return params;
-}
-
-// find the last Estimator or Pipeline that needs fit in
[GitHub] [flink] flinkbot edited a comment on pull request #17305: [hotfix][flink-avro] Removed unused variable
flinkbot edited a comment on pull request #17305: URL: https://github.com/apache/flink/pull/17305#issuecomment-921331062 ## CI report: * dfc9f4122197a6ebea6041cf39cdd2681c26bcdd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24310) * e962467ac5dfff2a6c6ed1f9b771435bc0a7a507 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18996) Avoid disorder for time interval join
[ https://issues.apache.org/jira/browse/FLINK-18996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18996: --- Labels: auto-deprioritized-major stale-critical (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Critical but is unassigned and neither itself nor its Sub-Tasks have been updated for 14 days. I have gone ahead and marked it "stale-critical". If this ticket is critical, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Avoid disorder for time interval join > - > > Key: FLINK-18996 > URL: https://issues.apache.org/jira/browse/FLINK-18996 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Reporter: Benchao Li >Priority: Critical > Labels: auto-deprioritized-major, stale-critical > > Currently, the time interval join will produce data with rowtime later than > watermark. If we use the rowtime again in downstream, e.t. window > aggregation, we'll lose some data. > > reported from user-zh: > [http://apache-flink.147419.n8.nabble.com/Re-flink-interval-join-tc4458.html#none] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19822) Remove redundant shuffle for streaming
[
https://issues.apache.org/jira/browse/FLINK-19822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-19822:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
(was: auto-unassigned pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Remove redundant shuffle for streaming
> --
>
> Key: FLINK-19822
> URL: https://issues.apache.org/jira/browse/FLINK-19822
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: godfrey he
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
>
> This is similar
> [FLINK-12575|https://issues.apache.org/jira/browse/FLINK-12575], we could
> implement {{satisfyTraits}} method for stream nodes to remove redundant
> shuffle. This could add more possibilities that more operators can be merged
> into multiple input operator.
> Different batch, stream operators require the shuffle keys and the state keys
> must be exactly the same, otherwise the state may be not correct.
> We only support a few operators in this issue, such as Join and regular
> Aggregate. Other operators will be supported in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-7151) Support function DDL
[
https://issues.apache.org/jira/browse/FLINK-7151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-7151:
--
Labels: auto-deprioritized-major auto-unassigned pull-request-available
(was: auto-unassigned pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> Support function DDL
>
>
> Key: FLINK-7151
> URL: https://issues.apache.org/jira/browse/FLINK-7151
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: yuemeng
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Based on create function and table.we can register a udf,udaf,udtf use sql:
> {code}
> CREATE FUNCTION [IF NOT EXISTS] [catalog_name.db_name.]function_name AS
> class_name;
> DROP FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name;
> ALTER FUNCTION [IF EXISTS] [catalog_name.db_name.]function_name RENAME TO
> new_name;
> {code}
> {code}
> CREATE function 'TOPK' AS
> 'com..aggregate.udaf.distinctUdaf.topk.ITopKUDAF';
> INSERT INTO db_sink SELECT id, TOPK(price, 5, 'DESC') FROM kafka_source GROUP
> BY id;
> {code}
> This ticket can assume that the function class is already loaded in classpath
> by users. Advanced syntax like to how to dynamically load udf libraries from
> external locations can be on a separate ticket.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * b73a10fcdf8bc9726d6a3a644ed535de6fc2557e Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24404) * 8e1e9761a59b5a271c9196fbc206e7dae4cea79b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24409) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17278: [FLINK-24280] Support manual checkpoints triggering from a MiniCluster
flinkbot edited a comment on pull request #17278: URL: https://github.com/apache/flink/pull/17278#issuecomment-919100526 ## CI report: * 395e3734a41ed95050f54ff955153d9d574ad3a6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24403) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 281fb59359cd937c0c84505888fdeedd7b002b88 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24401) * b73a10fcdf8bc9726d6a3a644ed535de6fc2557e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24404) * 8e1e9761a59b5a271c9196fbc206e7dae4cea79b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23976) Add additional availability timing metrics to Job lifecycle events
[ https://issues.apache.org/jira/browse/FLINK-23976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418765#comment-17418765 ] Seth Wiesman commented on FLINK-23976: -- [~thw] interesting. Similar to the linked ticket we'll need to think about how this fits into application mode, in particular jobs that multiple job graphs and which are potentially chained. > Add additional availability timing metrics to Job lifecycle events > -- > > Key: FLINK-23976 > URL: https://issues.apache.org/jira/browse/FLINK-23976 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: Seth Wiesman >Priority: Major > Labels: pull-request-available > > Flink currently contains a number of availability lifecycle metrics[1] > showing how long it takes to move through different job status'. We propose > adding two additional metrics; startingTime, and cancellingTime (open to > bikeshedding on the metric names). > > * startingTime is the time it takes a job to get to running. > * cancellingTime is the time spent in status CANCELLING > > > [1]https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/metrics/#availability -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] pnowojski commented on a change in pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
pnowojski commented on a change in pull request #17331:
URL: https://github.com/apache/flink/pull/17331#discussion_r714192215
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##
@@ -62,36 +115,45 @@ public CheckpointFailureManager(int
tolerableCpFailureNumber, FailJobCallback fa
* the failure happens before the checkpoint id generation. In this
case, it will be
* specified a negative latest generated checkpoint id as a special
flag.
*/
-public void handleJobLevelCheckpointException(
-CheckpointException exception, long checkpointId) {
-handleCheckpointException(exception, checkpointId,
failureCallback::failJob);
+void handleJobLevelCheckpointException(
+CheckpointProperties checkpointProperties,
+CheckpointException exception,
+long checkpointId) {
+if (!checkpointProperties.isSavepoint()) {
+checkFailureAgainstCounter(exception, checkpointId,
failureCallback::failJob);
+}
}
/**
* Handle task level checkpoint exception with a handler callback.
*
* @param exception the checkpoint exception.
- * @param checkpointId the failed checkpoint id used to count the
continuous failure number
+ * @param pendingCheckpoint the failed checkpoint used to count the
continuous failure number
* based on checkpoint id sequence. In trigger phase, we may not get
the checkpoint id when
* the failure happens before the checkpoint id generation. In this
case, it will be
* specified a negative latest generated checkpoint id as a special
flag.
* @param executionAttemptID the execution attempt id, as a safe guard.
Review comment:
nit: javadoc and parameters are out of order
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManagerTest.java
##
@@ -17,105 +17,137 @@
package org.apache.flink.runtime.checkpoint;
+import org.apache.flink.runtime.JobException;
import org.apache.flink.runtime.executiongraph.ExecutionAttemptID;
import org.apache.flink.util.TestLogger;
import org.junit.Test;
+import java.io.IOException;
+
import static
org.apache.flink.runtime.checkpoint.CheckpointFailureReason.CHECKPOINT_EXPIRED;
+import static
org.apache.flink.runtime.checkpoint.CheckpointProperties.forCheckpoint;
+import static
org.apache.flink.runtime.checkpoint.CheckpointRetentionPolicy.NEVER_RETAIN_AFTER_TERMINATION;
import static org.junit.Assert.assertEquals;
/** Tests for the checkpoint failure manager. */
public class CheckpointFailureManagerTest extends TestLogger {
@Test
-public void testIgnoresPastCheckpoints() {
+public void testIgnoresPastCheckpoints() throws IOException, JobException {
TestFailJobCallback callback = new TestFailJobCallback();
CheckpointFailureManager failureManager = new
CheckpointFailureManager(2, callback);
+CheckpointProperties checkpointProp =
forCheckpoint(NEVER_RETAIN_AFTER_TERMINATION);
Review comment:
`checkpointProperties` (and in other places)
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##
@@ -53,6 +56,56 @@ public CheckpointFailureManager(int
tolerableCpFailureNumber, FailJobCallback fa
this.countedCheckpointIds = ConcurrentHashMap.newKeySet();
}
+/**
+ * Failures on JM:
+ *
+ *
+ * all checkpoints - go against failure counter.
+ * any savepoints - don’t do anything, manual action, the failover
will not help anyway.
+ *
+ *
+ * Failures on TM:
+ *
+ *
+ * all checkpoints - go against failure counter (failover might help
and we want to notify
+ * users).
+ * sync savepoints - we must always fail, otherwise we risk deadlock
when the job
+ * cancelation waiting for finishing savepoint which never happens.
+ * non sync savepoints - go against failure counter (failover might
help solve the
+ * problem).
+ *
+ *
+ * @param pendingCheckpoint the failed checkpoint if it was initialized
already.
+ * @param checkpointProperties the checkpoint properties in order to
determinate which handle
+ * strategy can be used.
+ * @param exception the checkpoint exception.
+ * @param executionAttemptID the execution attempt id, as a safe guard.
+ */
+public void handleCheckpointException(
+@Nullable PendingCheckpoint pendingCheckpoint,
+CheckpointProperties checkpointProperties,
+CheckpointException exception,
+@Nullable ExecutionAttemptID executionAttemptID) {
+if (isJMFailure(pendingCheckpoint, exception, executionAttemptID)) {
+handleJobLevelCheckpointException(
+checkpointProperties,
+
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * cf1c7e94891826f0f4ce3edbb18def7b13f3483a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24402) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 281fb59359cd937c0c84505888fdeedd7b002b88 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24401) * b73a10fcdf8bc9726d6a3a644ed535de6fc2557e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24404) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17278: [FLINK-24280] Support manual checkpoints triggering from a MiniCluster
flinkbot edited a comment on pull request #17278: URL: https://github.com/apache/flink/pull/17278#issuecomment-919100526 ## CI report: * 870fe2769409248ca709d8967cc55a4a007fe8b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24351) * 395e3734a41ed95050f54ff955153d9d574ad3a6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24403) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17253: [FLINK-24182][task] Do not interrupt tasks/operators that are already closing
flinkbot edited a comment on pull request #17253: URL: https://github.com/apache/flink/pull/17253#issuecomment-917975072 ## CI report: * 43d1bfeb72746bdb502cfad17f189ad8410198ee Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24400) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 5465d4affc9ccb4c4d1fabaa4a7d909b091f5fc8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24399) * 281fb59359cd937c0c84505888fdeedd7b002b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24401) * b73a10fcdf8bc9726d6a3a644ed535de6fc2557e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17278: [FLINK-24280] Support manual checkpoints triggering from a MiniCluster
flinkbot edited a comment on pull request #17278: URL: https://github.com/apache/flink/pull/17278#issuecomment-919100526 ## CI report: * 870fe2769409248ca709d8967cc55a4a007fe8b3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24351) * 395e3734a41ed95050f54ff955153d9d574ad3a6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 5465d4affc9ccb4c4d1fabaa4a7d909b091f5fc8 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24399) * 281fb59359cd937c0c84505888fdeedd7b002b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24401) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23886) An exception is thrown out when recover job timers from checkpoint file
[ https://issues.apache.org/jira/browse/FLINK-23886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418400#comment-17418400 ] Steven Zhen Wu edited comment on FLINK-23886 at 9/22/21, 3:58 PM: -- Just to add another data point. We observed the same issue with Flink 1.13.2 in production. We don't know how to reproduce this type of tricky state corruption problem ``` org.apache.flink.util.FlinkRuntimeException: Error while deserializing the element. at org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.deserializeElement(RocksDBCachingPriorityQueueSet.java:388) at org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.peek(RocksDBCachingPriorityQueueSet.java:145) at org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.peek(RocksDBCachingPriorityQueueSet.java:58) at org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue$InternalPriorityQueueComparator.comparePriority(KeyGroupPartitionedPriorityQueue.java:285) at org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue$InternalPriorityQueueComparator.comparePriority(KeyGroupPartitionedPriorityQueue.java:271) at org.apache.flink.runtime.state.heap.HeapPriorityQueue.isElementPriorityLessThen(HeapPriorityQueue.java:161) at org.apache.flink.runtime.state.heap.HeapPriorityQueue.siftUp(HeapPriorityQueue.java:118) at org.apache.flink.runtime.state.heap.HeapPriorityQueue.addInternal(HeapPriorityQueue.java:82) at org.apache.flink.runtime.state.heap.AbstractHeapPriorityQueue.add(AbstractHeapPriorityQueue.java:71) at org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue.(KeyGroupPartitionedPriorityQueue.java:93) at org.apache.flink.contrib.streaming.state.RocksDBPriorityQueueSetFactory.create(RocksDBPriorityQueueSetFactory.java:113) at org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.create(RocksDBKeyedStateBackend.java:476) at org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.createTimerPriorityQueue(InternalTimeServiceManagerImpl.java:174) at org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.registerOrGetTimerService(InternalTimeServiceManagerImpl.java:160) at org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.getInternalTimerService(InternalTimeServiceManagerImpl.java:136) at org.apache.flink.streaming.api.operators.AbstractStreamOperator.getInternalTimerService(AbstractStreamOperator.java:620) at org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.open(WindowOperator.java:225) at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442) at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582) at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562) at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537) at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.io.EOFException at org.apache.flink.core.memory.DataInputDeserializer.readUnsignedByte(DataInputDeserializer.java:329) at org.apache.flink.types.StringValue.readString(StringValue.java:781) at org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:73) at org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:31) at org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:161) at org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:43) at org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.deserializeElement(RocksDBCachingPriorityQueueSet.java:386) ... 25 more ``` was (Author: stevenz3wu): Just to add another data point. We observed the same issue with Flink 1.13.2 in production. We don't know how to reproduce this type of tricky state corruption problem > An exception is thrown out when recover job timers from checkpoint file > --- > > Key: FLINK-23886 > URL: https://issues.apache.org/jira/browse/FLINK-23886 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.10.0, 1.11.3, 1.13.2 >Reporter: JING ZHANG >
[jira] [Comment Edited] (FLINK-23886) An exception is thrown out when recover job timers from checkpoint file
[
https://issues.apache.org/jira/browse/FLINK-23886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418400#comment-17418400
]
Steven Zhen Wu edited comment on FLINK-23886 at 9/22/21, 3:58 PM:
--
Just to add another data point. We observed the same issue with Flink 1.13.2 in
production. We don't know how to reproduce this type of tricky state corruption
problem
{code}
org.apache.flink.util.FlinkRuntimeException: Error while deserializing the
element.
at
org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.deserializeElement(RocksDBCachingPriorityQueueSet.java:388)
at
org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.peek(RocksDBCachingPriorityQueueSet.java:145)
at
org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.peek(RocksDBCachingPriorityQueueSet.java:58)
at
org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue$InternalPriorityQueueComparator.comparePriority(KeyGroupPartitionedPriorityQueue.java:285)
at
org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue$InternalPriorityQueueComparator.comparePriority(KeyGroupPartitionedPriorityQueue.java:271)
at
org.apache.flink.runtime.state.heap.HeapPriorityQueue.isElementPriorityLessThen(HeapPriorityQueue.java:161)
at
org.apache.flink.runtime.state.heap.HeapPriorityQueue.siftUp(HeapPriorityQueue.java:118)
at
org.apache.flink.runtime.state.heap.HeapPriorityQueue.addInternal(HeapPriorityQueue.java:82)
at
org.apache.flink.runtime.state.heap.AbstractHeapPriorityQueue.add(AbstractHeapPriorityQueue.java:71)
at
org.apache.flink.runtime.state.heap.KeyGroupPartitionedPriorityQueue.(KeyGroupPartitionedPriorityQueue.java:93)
at
org.apache.flink.contrib.streaming.state.RocksDBPriorityQueueSetFactory.create(RocksDBPriorityQueueSetFactory.java:113)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.create(RocksDBKeyedStateBackend.java:476)
at
org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.createTimerPriorityQueue(InternalTimeServiceManagerImpl.java:174)
at
org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.registerOrGetTimerService(InternalTimeServiceManagerImpl.java:160)
at
org.apache.flink.streaming.api.operators.InternalTimeServiceManagerImpl.getInternalTimerService(InternalTimeServiceManagerImpl.java:136)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.getInternalTimerService(AbstractStreamOperator.java:620)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.open(WindowOperator.java:225)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:582)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:562)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:647)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:537)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.EOFException
at
org.apache.flink.core.memory.DataInputDeserializer.readUnsignedByte(DataInputDeserializer.java:329)
at org.apache.flink.types.StringValue.readString(StringValue.java:781)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:73)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.deserialize(StringSerializer.java:31)
at
org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:161)
at
org.apache.flink.streaming.api.operators.TimerSerializer.deserialize(TimerSerializer.java:43)
at
org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.deserializeElement(RocksDBCachingPriorityQueueSet.java:386)
... 25 more
{code}
was (Author: stevenz3wu):
Just to add another data point. We observed the same issue with Flink 1.13.2 in
production. We don't know how to reproduce this type of tricky state corruption
problem
```
org.apache.flink.util.FlinkRuntimeException: Error while deserializing the
element.
at
org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.deserializeElement(RocksDBCachingPriorityQueueSet.java:388)
at
org.apache.flink.contrib.streaming.state.RocksDBCachingPriorityQueueSet.peek(RocksDBCachingPriorityQueueSet.java:145)
at
[GitHub] [flink] dawidwys commented on a change in pull request #17253: [FLINK-24182][task] Do not interrupt tasks/operators that are already closing
dawidwys commented on a change in pull request #17253:
URL: https://github.com/apache/flink/pull/17253#discussion_r713933683
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskCancellationTest.java
##
@@ -56,6 +60,41 @@
/** Tests for the StreamTask cancellation. */
public class StreamTaskCancellationTest extends TestLogger {
+@Test
+public void testDoNotInterruptWhileClosing() throws Exception {
+TestInterruptInCloseOperator testOperator = new
TestInterruptInCloseOperator();
+try (StreamTaskMailboxTestHarness harness =
+new
StreamTaskMailboxTestHarnessBuilder<>(OneInputStreamTask::new, STRING_TYPE_INFO)
+.addInput(STRING_TYPE_INFO)
+.setupOutputForSingletonOperatorChain(testOperator)
+.build()) {}
+}
+
+private static class TestInterruptInCloseOperator extends
AbstractStreamOperator
+implements OneInputStreamOperator {
+@Override
+public void close() throws Exception {
+super.close();
+
+AtomicBoolean running = new AtomicBoolean(true);
+Thread thread =
+new Thread(
+() -> {
+while (running.get()) {}
+});
+thread.start();
+try {
+getContainingTask().maybeInterruptOnCancel(thread, null, null);
Review comment:
just a comment: It is a very white box style of testing. The test
freezes the signature of `maybeInterruptOnCancel` and the internal flag
handling. However, given I don't have a better idea. I am fine with the test.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java
##
@@ -776,38 +776,14 @@ private void doRun() {
throw new CancelTaskException();
}
} catch (Throwable t) {
-
-// unwrap wrapped exceptions to make stack traces more compact
-if (t instanceof WrappingRuntimeException) {
-t = ((WrappingRuntimeException) t).unwrap();
-}
+t = preProcessException(t);
Review comment:
nit: move below the block comment
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskCancellationTest.java
##
@@ -95,31 +90,6 @@ public void close() throws Exception {
public void processElement(StreamRecord element) throws
Exception {}
}
-@Test
-public void testCancellationWaitsForActiveTimers() throws Exception {
-StreamTaskWithBlockingTimer.reset();
-ResultPartitionDeploymentDescriptor descriptor =
-new ResultPartitionDeploymentDescriptor(
-PartitionDescriptorBuilder.newBuilder().build(),
-
NettyShuffleDescriptorBuilder.newBuilder().buildLocal(),
-1,
-false);
-Task task =
-new TestTaskBuilder(new
NettyShuffleEnvironmentBuilder().build())
-.setInvokable(StreamTaskWithBlockingTimer.class)
-.setResultPartitions(singletonList(descriptor))
-.build();
-task.startTaskThread();
-
-StreamTaskWithBlockingTimer.timerStarted.join();
Review comment:
`StreamTaskWithBlockingTimer` is now unused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai commented on pull request #271: [FLINK-24356] Support delayed message cancellation in the remote SDKs
tzulitai commented on pull request #271: URL: https://github.com/apache/flink-statefun/pull/271#issuecomment-925038097 +1, LGTM. Not as part of this PR, and could probably be hard due to its indeterministic / best-effort nature of cancellations, but I think we thought of adding delayed message cancellation to the E2E tests. Do you think a JIRA should be opened specifically for that before merging this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24031) I am trying to deploy Flink in kubernetes but when I launch the taskManager in other container I get a Exception
[
https://issues.apache.org/jira/browse/FLINK-24031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julio Pérez updated FLINK-24031:
Priority: Major (was: Minor)
> I am trying to deploy Flink in kubernetes but when I launch the taskManager
> in other container I get a Exception
>
>
> Key: FLINK-24031
> URL: https://issues.apache.org/jira/browse/FLINK-24031
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.13.0, 1.13.2
>Reporter: Julio Pérez
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.1
>
> Attachments: flink-map.yml, jobmanager.log, jobmanager.yml,
> taskmanager.log, taskmanager.yml
>
>
> I explain here -> [https://github.com/apache/flink/pull/17020]
> I have a problem when I try to run Flink in k8s with the follow manifests
> I have the following exception
> # JobManager :
> {quote}2021-08-27 09:16:57,917 ERROR akka.remote.EndpointWriter [] - dropping
> message [class akka.actor.ActorSelectionMessage] for non-local recipient
> [Actor[akka.tcp://flink@jobmanager-hs:6123/]] arriving at
> [akka.tcp://flink@jobmanager-hs:6123] inbound addresses are
> [akka.tcp://flink@cluster:6123]
> 2021-08-27 09:17:01,255 DEBUG
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Trigger heartbeat request.
> 2021-08-27 09:17:01,284 DEBUG
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] -
> Trigger heartbeat request.
> 2021-08-27 09:17:10,008 DEBUG akka.remote.transport.netty.NettyTransport []
> - Remote connection to [/172.17.0.1:34827] was disconnected because of [id:
> 0x13ae1d03, /172.17.0.1:34827 :> /172.17.0.23:6123] DISCONNECTED
> 2021-08-27 09:17:10,008 DEBUG akka.remote.transport.ProtocolStateActor [] -
> Association between local [tcp://flink@cluster:6123] and remote
> [tcp://flink@172.17.0.1:34827] was disassociated because the
> ProtocolStateActor failed: Unknown
> 2021-08-27 09:17:10,009 WARN akka.remote.ReliableDeliverySupervisor [] -
> Association with remote system [akka.tcp://flink@172.17.0.24:6122] has
> failed, address is now gated for [50] ms. Reason: [Disassociated]
> {quote}
> TaskManager:
> {quote}INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not
> resolve ResourceManager address
> akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager__, retrying
> in 1 ms: Could not connect to rpc endpoint under address
> akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager__.
> INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not
> resolve ResourceManager address
> akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager__, retrying
> in 1 ms: Could not connect to rpc endpoint under address
> akka.tcp://flink@flink-jobmanager:6123/user/rpc/resourcemanager__.
> {quote}
> Best regards,
> Julio
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
[
https://issues.apache.org/jira/browse/FLINK-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418658#comment-17418658
]
Piotr Nowojski commented on FLINK-24357:
Very unlikely, but potentially related to FLINK-24182, as it was first spotted
in a PR for this issue.
> ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> fails with an Unhandled error
> -
>
> Key: FLINK-24357
> URL: https://issues.apache.org/jira/browse/FLINK-24357
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Piotr Nowojski
>Priority: Major
>
> In a [private
> azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
> build when testing my own PR I've noticed the following error that looks
> unrelated to any of my changes (modifications to {{Task}} class
> error/cancellation handling logic):
> {noformat}
> 2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
> testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
> Time elapsed: 28.753 s <<< FAILURE!
> 2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
> TestingFatalErrorHandler caught an exception.
> 2021-09-22T08:09:16.6246513Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
> 2021-09-22T08:09:16.6247281Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
> 2021-09-22T08:09:16.6248167Z Sep 22 08:09:16 at
> org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
> 2021-09-22T08:09:16.6248862Z Sep 22 08:09:16 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2021-09-22T08:09:16.6249620Z Sep 22 08:09:16 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-09-22T08:09:16.6250210Z Sep 22 08:09:16 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2021-09-22T08:09:16.6250773Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6251375Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2021-09-22T08:09:16.6251951Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2021-09-22T08:09:16.6252562Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2021-09-22T08:09:16.6253415Z Sep 22 08:09:16 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2021-09-22T08:09:16.6254469Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2021-09-22T08:09:16.6255039Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2021-09-22T08:09:16.6256238Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2021-09-22T08:09:16.6257109Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2021-09-22T08:09:16.6257766Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2021-09-22T08:09:16.6258406Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2021-09-22T08:09:16.6259050Z Sep 22 08:09:16 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2021-09-22T08:09:16.6259827Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> 2021-09-22T08:09:16.6260963Z Sep 22 08:09:16 at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> 2021-09-22T08:09:16.6261796Z Sep 22 08:09:16 at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
> 2021-09-22T08:09:16.6262428Z Sep 22 08:09:16 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> 2021-09-22T08:09:16.6263268Z Sep 22 08:09:16 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> 2021-09-22T08:09:16.6263875Z Sep 22 08:09:16 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> 2021-09-22T08:09:16.6265025Z Sep 22 08:09:16 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> 2021-09-22T08:09:16.6265940Z Sep 22 08:09:16 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> 2021-09-22T08:09:16.6266767Z Sep 22 08:09:16 at
>
[jira] [Created] (FLINK-24357) ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled fails with an Unhandled error
Piotr Nowojski created FLINK-24357:
--
Summary:
ZooKeeperLeaderElectionConnectionHandlingTest#testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled
fails with an Unhandled error
Key: FLINK-24357
URL: https://issues.apache.org/jira/browse/FLINK-24357
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Piotr Nowojski
In a [private
azure|https://dev.azure.com/pnowojski/637f6470-2732-4605-a304-caebd40e284b/_apis/build/builds/517/logs/155]
build when testing my own PR I've noticed the following error that looks
unrelated to any of my changes (modifications to {{Task}} class
error/cancellation handling logic):
{noformat}
2021-09-22T08:09:16.6244936Z Sep 22 08:09:16 [ERROR]
testLoseLeadershipOnLostConnectionIfTolerateSuspendedConnectionsIsEnabled Time
elapsed: 28.753 s <<< FAILURE!
2021-09-22T08:09:16.6245821Z Sep 22 08:09:16 java.lang.AssertionError: The
TestingFatalErrorHandler caught an exception.
2021-09-22T08:09:16.6246513Z Sep 22 08:09:16at
org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.after(TestingFatalErrorHandlerResource.java:78)
2021-09-22T08:09:16.6247281Z Sep 22 08:09:16at
org.apache.flink.runtime.util.TestingFatalErrorHandlerResource.access$300(TestingFatalErrorHandlerResource.java:33)
2021-09-22T08:09:16.6248167Z Sep 22 08:09:16at
org.apache.flink.runtime.util.TestingFatalErrorHandlerResource$1.evaluate(TestingFatalErrorHandlerResource.java:57)
2021-09-22T08:09:16.6248862Z Sep 22 08:09:16at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2021-09-22T08:09:16.6249620Z Sep 22 08:09:16at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2021-09-22T08:09:16.6250210Z Sep 22 08:09:16at
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
2021-09-22T08:09:16.6250773Z Sep 22 08:09:16at
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2021-09-22T08:09:16.6251375Z Sep 22 08:09:16at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
2021-09-22T08:09:16.6251951Z Sep 22 08:09:16at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
2021-09-22T08:09:16.6252562Z Sep 22 08:09:16at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
2021-09-22T08:09:16.6253415Z Sep 22 08:09:16at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
2021-09-22T08:09:16.6254469Z Sep 22 08:09:16at
org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2021-09-22T08:09:16.6255039Z Sep 22 08:09:16at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
2021-09-22T08:09:16.6256238Z Sep 22 08:09:16at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
2021-09-22T08:09:16.6257109Z Sep 22 08:09:16at
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
2021-09-22T08:09:16.6257766Z Sep 22 08:09:16at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
2021-09-22T08:09:16.6258406Z Sep 22 08:09:16at
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2021-09-22T08:09:16.6259050Z Sep 22 08:09:16at
org.junit.runners.ParentRunner.run(ParentRunner.java:413)
2021-09-22T08:09:16.6259827Z Sep 22 08:09:16at
org.junit.runner.JUnitCore.run(JUnitCore.java:137)
2021-09-22T08:09:16.6260963Z Sep 22 08:09:16at
org.junit.runner.JUnitCore.run(JUnitCore.java:115)
2021-09-22T08:09:16.6261796Z Sep 22 08:09:16at
org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
2021-09-22T08:09:16.6262428Z Sep 22 08:09:16at
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
2021-09-22T08:09:16.6263268Z Sep 22 08:09:16at
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
2021-09-22T08:09:16.6263875Z Sep 22 08:09:16at
java.util.Iterator.forEachRemaining(Iterator.java:116)
2021-09-22T08:09:16.6265025Z Sep 22 08:09:16at
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
2021-09-22T08:09:16.6265940Z Sep 22 08:09:16at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
2021-09-22T08:09:16.6266767Z Sep 22 08:09:16at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
2021-09-22T08:09:16.6267470Z Sep 22 08:09:16at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
2021-09-22T08:09:16.6268165Z Sep 22 08:09:16at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
2021-09-22T08:09:16.6269341Z Sep 22 08:09:16at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2021-09-22T08:09:16.6269928Z Sep 22 08:09:16at
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 66b303dcd4c117307248cc2f80f2e50733b08b0e Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24395) * 5465d4affc9ccb4c4d1fabaa4a7d909b091f5fc8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24399) * 281fb59359cd937c0c84505888fdeedd7b002b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24401) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN * 765eb79602d0d359ad2267f97cc5785663016c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24398) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * 8aebff6d6408a9210dcf95342eb31df8f9e1fee3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24341) * cf1c7e94891826f0f4ce3edbb18def7b13f3483a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24402) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] matriv commented on a change in pull request #17313: [FLINK-16204][table] Support JSON_ARRAY()
matriv commented on a change in pull request #17313:
URL: https://github.com/apache/flink/pull/17313#discussion_r714024229
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/inference/strategies/SpecificInputTypeStrategies.java
##
@@ -76,6 +77,18 @@
logical(LogicalTypeRoot.BOOLEAN),
logical(LogicalTypeFamily.NUMERIC;
+/** Input strategy for {@link BuiltInFunctionDefinitions#JSON_ARRAY}. */
+public static final InputTypeStrategy JSON_ARRAY =
+varyingSequence(
+symbol(JsonOnNull.class),
+or(
+logical(LogicalTypeFamily.CHARACTER_STRING),
+logical(LogicalTypeFamily.BINARY_STRING),
+logical(LogicalTypeFamily.TIMESTAMP),
+logical(LogicalTypeFamily.CONSTRUCTED),
+logical(LogicalTypeRoot.BOOLEAN),
+logical(LogicalTypeFamily.NUMERIC)));
Review comment:
I mean in the places where you list some examples, like here:
https://github.com/apache/flink/pull/17313/files#diff-539fb22ee6aeee4cf07230bb4155500c6680c4cc889260e2c58bfa9d63fb7de5R775
to also include an example of an array with nested json objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
flinkbot edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-904541061 ## CI report: * 8aebff6d6408a9210dcf95342eb31df8f9e1fee3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24341) * cf1c7e94891826f0f4ce3edbb18def7b13f3483a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
pnowojski commented on a change in pull request #17331:
URL: https://github.com/apache/flink/pull/17331#discussion_r713977586
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointFailureManager.java
##
@@ -53,6 +56,49 @@ public CheckpointFailureManager(int
tolerableCpFailureNumber, FailJobCallback fa
this.countedCheckpointIds = ConcurrentHashMap.newKeySet();
}
+/**
+ * Failures on JM:
+ *
+ *
+ * all checkpoints - go against failure counter.
+ * any savepoints - don’t do anything, manual action, the failover
will not help anyway.
+ *
+ *
+ * Failures on TM:
+ *
+ *
+ * all checkpoints - go against failure counter (failover might help
and we want to notify
+ * users).
+ * sync savepoints - we must always fail, otherwise we risk deadlock
when the job
+ * cancelation waiting for finishing savepoint which never happens.
+ * non sync savepoints - go against failure counter (failover might
help solve the
+ * problem).
+ *
+ *
+ * @param pendingCheckpoint the failed checkpoint if it was initialized
already.
+ * @param checkpointProperties the checkpoint properties in order to
determinate which handle
+ * strategy can be used.
+ * @param exception the checkpoint exception.
+ * @param executionAttemptID the execution attempt id, as a safe guard.
+ */
+public void handleCheckpointException(
+@Nullable PendingCheckpoint pendingCheckpoint,
+CheckpointProperties checkpointProperties,
+CheckpointException exception,
+@Nullable ExecutionAttemptID executionAttemptID) {
+if (checkpointProperties.isSavepoint() &&
checkpointProperties.isSynchronous()) {
+handleSynchronousSavepointFailure(exception);
+} else if (executionAttemptID != null && pendingCheckpoint != null) {
+handleTaskLevelCheckpointException(
+exception, pendingCheckpoint.getCheckpointId(),
executionAttemptID);
+} else if (pendingCheckpoint != null) {
+handleJobLevelCheckpointException(exception,
pendingCheckpoint.getCheckpointId());
+} else if (!checkpointProperties.isSavepoint()) {
+handleJobLevelCheckpointException(exception);
+}
Review comment:
I think it's quite difficult to follow the logic/conditions here. Can we
maybe rewrite it to mimic the javadoc logic? Split the logic TM/JM failure
first, next whether it is checkpoint/savepoint/sync savepoint. And we could
also rename the methods called from here, to things like:
```
checkFailureAgainstCounter();
failJob();
```
?
For example as it is, I was quite confused where do we handle
> any savepoints - don’t do anything, manual action, the failover will
not help anyway.
and it's actually not in this method, but one level deeper.
If this method grows too large, we could always split it:
```
if (isTMFailure()) {
handleTaskLevelCheckpointException(...);
}
else {
handleJobLevelCheckpointException(...);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17335: [FLINK-24353][scripts] Respect dynamic configurations when calculating memory sizes.
flinkbot edited a comment on pull request #17335: URL: https://github.com/apache/flink/pull/17335#issuecomment-924765839 ## CI report: * c95b8b78c37bc2dd4580bd245dc99a939b3927d8 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24393) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 336d7f0aca4d20c49c1f6cb95ac3702511e8033c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24363) * 66b303dcd4c117307248cc2f80f2e50733b08b0e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24395) * 5465d4affc9ccb4c4d1fabaa4a7d909b091f5fc8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24399) * 281fb59359cd937c0c84505888fdeedd7b002b88 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24401) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-23189) Count and fail the task when the disk is error on JobManager
[ https://issues.apache.org/jira/browse/FLINK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418602#comment-17418602 ] zlzhang0122 edited comment on FLINK-23189 at 9/22/21, 1:52 PM: --- [~pnowojski] ok, I've seen the fix and found that it added the handled of the onTriggerFailure when the checkpoint is null, I've noticed this situation but I didn't reproduced it in our production environment, so I didn't change the code here, but actually we may indeed need this fix for this case. was (Author: zlzhang0122): [~pnowojski] ok, I've seen the fix and found that it added the handled of the onTriggerFailure when the checkpoint is null, I've found this situation but I didn't reproduced it in our production environment, so I didn't change the code here, but actually we may indeed need that fix for some corner cases. > Count and fail the task when the disk is error on JobManager > > > Key: FLINK-23189 > URL: https://issues.apache.org/jira/browse/FLINK-23189 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.12.2, 1.13.1 >Reporter: zlzhang0122 >Assignee: zlzhang0122 >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: exception.txt > > > When the jobmanager disk is error and the triggerCheckpoint will throw a > IOException and fail, this will cause a TRIGGER_CHECKPOINT_FAILURE, but this > failure won't cause Job failed. Users can hardly find this error if he don't > see the JobManager logs. To avoid this case, I propose that we can figure out > these IOException case and increase the failureCounter which can fail the job > finally. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24338) Provide an interface to validate flinksql
[ https://issues.apache.org/jira/browse/FLINK-24338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418611#comment-17418611 ] Martijn Visser commented on FLINK-24338: I don't think that will work since Flink SQL implements Calcite. Would something like the [Flink SQL Gateway|https://github.com/ververica/flink-sql-gateway] be interesting? > Provide an interface to validate flinksql > - > > Key: FLINK-24338 > URL: https://issues.apache.org/jira/browse/FLINK-24338 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Spongebob >Priority: Minor > > It would be great if there is an interface that can validate flinksql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24338) Provide an interface to validate flinksql
[ https://issues.apache.org/jira/browse/FLINK-24338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418607#comment-17418607 ] Ingo Bürk commented on FLINK-24338: --- For a basic yes/no answer, the EXPLAIN solution would probably work well. I agree that we need a proper validation mechanism to allow building tooling around Flink SQL. Proper validation should also return detailed error details such as Line/column number when possible. There's also different levels of validation, i.e. purely on the syntax level, or including logical validation of whether a table exists, or can even be instantiated, same for functions etc. We'd probably have to expose these different levels separately as validating the existence of a table might not always be desired. There's also the question of validating several statements at once, eg some DDL to create a table and then a query on that table. This is even trickier because the validation of the query depends on the side effects of the first statement. Even purely syntactical validation using the parser currently isn't possible due to side effects performed in the parser. This is a technical debt that would have to be resolved and requires changing the design of many Operation subclasses. > Provide an interface to validate flinksql > - > > Key: FLINK-24338 > URL: https://issues.apache.org/jira/browse/FLINK-24338 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Spongebob >Priority: Minor > > It would be great if there is an interface that can validate flinksql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24338) Provide an interface to validate flinksql
[ https://issues.apache.org/jira/browse/FLINK-24338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418603#comment-17418603 ] Spongebob commented on FLINK-24338: --- Hi Patrick, I'd like to ask that how can I validate FlinkSQL with calcite? If you could give me some sample that would be great. > Provide an interface to validate flinksql > - > > Key: FLINK-24338 > URL: https://issues.apache.org/jira/browse/FLINK-24338 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Spongebob >Priority: Minor > > It would be great if there is an interface that can validate flinksql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Airblader commented on a change in pull request #17332: [FLINK-24349]Support customized Calalogs via JDBC
Airblader commented on a change in pull request #17332: URL: https://github.com/apache/flink/pull/17332#discussion_r713954142 ## File path: flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/JdbcDialect.java ## @@ -142,4 +143,16 @@ */ String getSelectFromStatement( String tableName, String[] selectFields, String[] conditionFields); + +/** Create catalog instance. */ +default AbstractJdbcCatalog createCatalog( +String catalogName, Review comment: Maybe CC @bowenli86 and @JingsongLi to give feedback on this, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17332: [FLINK-24349]Support customized Calalogs via JDBC
Airblader commented on a change in pull request #17332: URL: https://github.com/apache/flink/pull/17332#discussion_r713951668 ## File path: flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/JdbcDialect.java ## @@ -142,4 +143,16 @@ */ String getSelectFromStatement( String tableName, String[] selectFields, String[] conditionFields); + +/** Create catalog instance. */ +default AbstractJdbcCatalog createCatalog( +String catalogName, Review comment: Basically, I think we should just pass the `context` argument from `JdbcCatalogFactory#createCatalog` directly to `JdbcDialect#createCatalog` instead of "destructuring" it into into a list of the individual options. Then any changes to `CatalogFactory#Context` would also automatically become available for dialect implementations. I think we should also remove `username`, `pwd`, `baseUrl`, `defaultUrl`, `getUsername`, `getPassword`, and `getBaseUrl` from `AbstractJdbcCatalog`. Instead add a `abstract Connection getConnection()` method to `AbstractJdbcCatalog` and use that in `#open`. This decouples the catalog implementation from the concrete authentication implementation. But I'm happy to hear other people's thoughts on this, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17332: [FLINK-24349]Support customized Calalogs via JDBC
Airblader commented on a change in pull request #17332: URL: https://github.com/apache/flink/pull/17332#discussion_r713951668 ## File path: flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/JdbcDialect.java ## @@ -142,4 +143,16 @@ */ String getSelectFromStatement( String tableName, String[] selectFields, String[] conditionFields); + +/** Create catalog instance. */ +default AbstractJdbcCatalog createCatalog( +String catalogName, Review comment: Basically, I think we should just pass the `context` argument from `JdbcCatalogFactory#createCatalog` directly to `JdbcDialect#createCatalog` instead of "destructuring" it into into a list of the individual options. Then any changes to `CatalogFactory#Context` would also automatically become available for dialect implementations. I think we should also remove `username`, `pwd`, `baseUrl`, `defaultUrl`, `getUsername`, `getPassword`, and `getBaseUrl` from `AbstractJdbcCatalog`. We can either move the connection test logic of `#open` into `JdbcCatalogUtils` and call it from `PostgresCatalog`, or we instead add a `abstract Connection getConnection()` method to `AbstractJdbcCatalog` and use that in `#open`. This decouples the catalog implementation from the concrete authentication implementation. But I'm happy to hear other people's thoughts on this, too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17332: [FLINK-24349]Support customized Calalogs via JDBC
Airblader commented on a change in pull request #17332: URL: https://github.com/apache/flink/pull/17332#discussion_r713951668 ## File path: flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/dialect/JdbcDialect.java ## @@ -142,4 +143,16 @@ */ String getSelectFromStatement( String tableName, String[] selectFields, String[] conditionFields); + +/** Create catalog instance. */ +default AbstractJdbcCatalog createCatalog( +String catalogName, Review comment: Basically, I think we should just pass the `context` argument from `JdbcCatalogFactory#createCatalog` directly to `JdbcDialect#createCatalog` instead of "destructuring" it into into a list of the individual options. Then any changes to `CatalogFactory#Context` would also automatically become available for dialect implementations. I think we should also remove `username`, `pwd`, `baseUrl`, `defaultUrl`, `getUsername`, `getPassword`, and `getBaseUrl` from `AbstractJdbcCatalog`. We can either move the connection test logic of `#open` into `JdbcCatalogUtils` and call it from `PostgresCatalog`, or we instead add a `abstract Connection getConnection()` method to `AbstractJdbcCatalog` and use that in `#open`. This decouples the catalog implementation from the concrete authentication implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23189) Count and fail the task when the disk is error on JobManager
[ https://issues.apache.org/jira/browse/FLINK-23189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418602#comment-17418602 ] zlzhang0122 commented on FLINK-23189: - [~pnowojski] ok, I've seen the fix and found that it added the handled of the onTriggerFailure when the checkpoint is null, I've found this situation but I didn't reproduced it in our production environment, so I didn't change the code here, but actually we may indeed need that fix for some corner cases. > Count and fail the task when the disk is error on JobManager > > > Key: FLINK-23189 > URL: https://issues.apache.org/jira/browse/FLINK-23189 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.12.2, 1.13.1 >Reporter: zlzhang0122 >Assignee: zlzhang0122 >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > Attachments: exception.txt > > > When the jobmanager disk is error and the triggerCheckpoint will throw a > IOException and fail, this will cause a TRIGGER_CHECKPOINT_FAILURE, but this > failure won't cause Job failed. Users can hardly find this error if he don't > see the JobManager logs. To avoid this case, I propose that we can figure out > these IOException case and increase the failureCounter which can fail the job > finally. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24356) Not able to cancel delayed message using remote stateful function mode
[ https://issues.apache.org/jira/browse/FLINK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418601#comment-17418601 ] Julia Pedrycz commented on FLINK-24356: --- I see PR, I will try to build and test it. > Not able to cancel delayed message using remote stateful function mode > -- > > Key: FLINK-24356 > URL: https://issues.apache.org/jira/browse/FLINK-24356 > Project: Flink > Issue Type: Bug > Components: Build System / Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Julia Pedrycz >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > > My use case is processing delayed messages with ability to cancel them and > update their delay. Whole logic is to wait with sending messages from ingress > to egress. > At first I worked with embedded mode, with > org.apache.flink.statefun.sdk.Context. Just for testing I’ve made a function > that invoked cancelDelayedMessage() method and straight after sendAfter() > method. I used message id as cancellation token. My function worked just > fine, message was overridden (cancelled and send with new delay). > After that I wanted to switch to remote mode, so I used > org.apache.flink.statefun.sdk.java.Context. I’ve made similar function as > previously, but it did not work. Just to be sure that cancellation works, I > invoked sendAfter() and straight after that cancelDelayedMessage() method. > Messages were still appearing on egress topic after original delay. > Questions: > 1. Whether cancelDelayedMessage() is working in remote mode? > 2. Should it work the same like in embedded mode? > 3. Should it work as expected if we call cancelDelayedMessage() in a moment > after sendAfter()? (for testing purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 336d7f0aca4d20c49c1f6cb95ac3702511e8033c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24363) * 66b303dcd4c117307248cc2f80f2e50733b08b0e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24395) * 5465d4affc9ccb4c4d1fabaa4a7d909b091f5fc8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24399) * 281fb59359cd937c0c84505888fdeedd7b002b88 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17253: [FLINK-24182][task] Do not interrupt tasks/operators that are already closing
flinkbot edited a comment on pull request #17253: URL: https://github.com/apache/flink/pull/17253#issuecomment-917975072 ## CI report: * a9a487d8ce9a9112f67e42de61c7eb249bd2ea58 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24388) * 43d1bfeb72746bdb502cfad17f189ad8410198ee Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24400) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 336d7f0aca4d20c49c1f6cb95ac3702511e8033c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24363) * 66b303dcd4c117307248cc2f80f2e50733b08b0e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24395) * 5465d4affc9ccb4c4d1fabaa4a7d909b091f5fc8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24399) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN * 765eb79602d0d359ad2267f97cc5785663016c2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24398) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17253: [FLINK-24182][task] Do not interrupt tasks/operators that are already closing
flinkbot edited a comment on pull request #17253: URL: https://github.com/apache/flink/pull/17253#issuecomment-917975072 ## CI report: * a9a487d8ce9a9112f67e42de61c7eb249bd2ea58 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24388) * 43d1bfeb72746bdb502cfad17f189ad8410198ee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24168) Rowtime type is not correct for windowTableFunction or OverAggregate follows after Match because the output type does not updated after input rowtime attribute changed
[
https://issues.apache.org/jira/browse/FLINK-24168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418568#comment-17418568
]
Leonard Xu commented on FLINK-24168:
[~qingru zhang] [~godfreyhe] Should we also add documentation for 1.14 and
1.13 ?
> Rowtime type is not correct for windowTableFunction or OverAggregate follows
> after Match because the output type does not updated after input rowtime
> attribute changed from rowtime to rowtime_ltz
> ---
>
> Key: FLINK-24168
> URL: https://issues.apache.org/jira/browse/FLINK-24168
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.14.0
>Reporter: JING ZHANG
>Assignee: JING ZHANG
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> Rowtime type is not correct for windowTableFunction or OverAggregate on Match
> because the output type does not updated after input rowtime attribute
> changed from rowtime to rowtime_ltz in `RelTimeIndicator`.
> The bug could be reproduced by the following two cases:
> {code:java}
> @Test
> def testWindowTVFOnMatchRecognizeOnRowtimeLTZ(): Unit = {
> val sqlQuery =
> s"""
>|SELECT
>| *
>|FROM Ticker
>|MATCH_RECOGNIZE (
>| PARTITION BY symbol
>| ORDER BY ts_ltz
>| MEASURES
>|A.price as price,
>|A.tax as tax,
>|MATCH_ROWTIME() as matchRowtime
>| ONE ROW PER MATCH
>| PATTERN (A)
>| DEFINE
>|A AS A.price > 0
>|) AS T
>|""".stripMargin
> val table = util.tableEnv.sqlQuery(sqlQuery)
> util.tableEnv.registerTable("T", table)
> val sqlQuery1 =
> s"""
>|SELECT *
>|FROM TABLE(TUMBLE(TABLE T, DESCRIPTOR(matchRowtime), INTERVAL '3'
> second))
>|""".stripMargin
> util.verifyRelPlanWithType(sqlQuery1)
> }
> @Test
> def testOverWindowOnMatchRecognizeOnRowtimeLTZ(): Unit = {
> val sqlQuery =
> s"""
>|SELECT
>| *
>|FROM Ticker
>|MATCH_RECOGNIZE (
>| PARTITION BY symbol
>| ORDER BY ts_ltz
>| MEASURES
>|A.price as price,
>|A.tax as tax,
>|MATCH_ROWTIME() as matchRowtime
>| ONE ROW PER MATCH
>| PATTERN (A)
>| DEFINE
>|A AS A.price > 0
>|) AS T
>|""".stripMargin
> val table = util.tableEnv.sqlQuery(sqlQuery)
> util.tableEnv.registerTable("T", table)
> val sqlQuery1 =
> """
> |SELECT
> | symbol,
> | price,
> | tax,
> | matchRowtime,
> | SUM(price) OVER (
> |PARTITION BY symbol ORDER BY matchRowtime RANGE UNBOUNDED
> PRECEDING) as price_sum
> |FROM T
> """.stripMargin
> util.verifyRelPlanWithType(sqlQuery1)
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24356) Not able to cancel delayed message using remote stateful function mode
[ https://issues.apache.org/jira/browse/FLINK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-24356: --- Labels: pull-request-available (was: ) > Not able to cancel delayed message using remote stateful function mode > -- > > Key: FLINK-24356 > URL: https://issues.apache.org/jira/browse/FLINK-24356 > Project: Flink > Issue Type: Bug > Components: Build System / Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Julia Pedrycz >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > > My use case is processing delayed messages with ability to cancel them and > update their delay. Whole logic is to wait with sending messages from ingress > to egress. > At first I worked with embedded mode, with > org.apache.flink.statefun.sdk.Context. Just for testing I’ve made a function > that invoked cancelDelayedMessage() method and straight after sendAfter() > method. I used message id as cancellation token. My function worked just > fine, message was overridden (cancelled and send with new delay). > After that I wanted to switch to remote mode, so I used > org.apache.flink.statefun.sdk.java.Context. I’ve made similar function as > previously, but it did not work. Just to be sure that cancellation works, I > invoked sendAfter() and straight after that cancelDelayedMessage() method. > Messages were still appearing on egress topic after original delay. > Questions: > 1. Whether cancelDelayedMessage() is working in remote mode? > 2. Should it work the same like in embedded mode? > 3. Should it work as expected if we call cancelDelayedMessage() in a moment > after sendAfter()? (for testing purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] igalshilman opened a new pull request #271: [FLINK-24356] Support delayed message cancellation in the remote SDKs
igalshilman opened a new pull request #271: URL: https://github.com/apache/flink-statefun/pull/271 This PR fixes a bug in a delayed message cancellation flow for remote SDKS. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 336d7f0aca4d20c49c1f6cb95ac3702511e8033c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24363) * 66b303dcd4c117307248cc2f80f2e50733b08b0e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24395) * 5465d4affc9ccb4c4d1fabaa4a7d909b091f5fc8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 1d20e8536f5cb590b904763fbcec1a5a2d074164 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24246) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24396) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24377) * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN * 765eb79602d0d359ad2267f97cc5785663016c2d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on pull request #17328: Group of hotfix commits fixing a couple of things in the tests
pnowojski commented on pull request #17328: URL: https://github.com/apache/flink/pull/17328#issuecomment-924871370 [private azure build is green](https://dev.azure.com/pnowojski/Flink/_build/results?buildId=516=results), merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski merged pull request #17328: Group of hotfix commits fixing a couple of things in the tests
pnowojski merged pull request #17328: URL: https://github.com/apache/flink/pull/17328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on pull request #13081: [FLINK-18590][json] Support json array explode to multi messages
slinkydeveloper commented on pull request #13081: URL: https://github.com/apache/flink/pull/13081#issuecomment-924865986 Hi @libenchao are you still interested in working on this issue? Can you rebase? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24356) Not able to cancel delayed message using remote stateful function mode
[ https://issues.apache.org/jira/browse/FLINK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418559#comment-17418559 ] Igal Shilman commented on FLINK-24356: -- [~Pedrycz] I have forgot to ask, which remote SDK are you using? > Not able to cancel delayed message using remote stateful function mode > -- > > Key: FLINK-24356 > URL: https://issues.apache.org/jira/browse/FLINK-24356 > Project: Flink > Issue Type: Bug > Components: Build System / Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Julia Pedrycz >Assignee: Igal Shilman >Priority: Major > > My use case is processing delayed messages with ability to cancel them and > update their delay. Whole logic is to wait with sending messages from ingress > to egress. > At first I worked with embedded mode, with > org.apache.flink.statefun.sdk.Context. Just for testing I’ve made a function > that invoked cancelDelayedMessage() method and straight after sendAfter() > method. I used message id as cancellation token. My function worked just > fine, message was overridden (cancelled and send with new delay). > After that I wanted to switch to remote mode, so I used > org.apache.flink.statefun.sdk.java.Context. I’ve made similar function as > previously, but it did not work. Just to be sure that cancellation works, I > invoked sendAfter() and straight after that cancelDelayedMessage() method. > Messages were still appearing on egress topic after original delay. > Questions: > 1. Whether cancelDelayedMessage() is working in remote mode? > 2. Should it work the same like in embedded mode? > 3. Should it work as expected if we call cancelDelayedMessage() in a moment > after sendAfter()? (for testing purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (FLINK-24356) Not able to cancel delayed message using remote stateful function mode
[ https://issues.apache.org/jira/browse/FLINK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igal Shilman updated FLINK-24356: - Comment: was deleted (was: [~Pedrycz] I have forgot to ask, which remote SDK are you using?) > Not able to cancel delayed message using remote stateful function mode > -- > > Key: FLINK-24356 > URL: https://issues.apache.org/jira/browse/FLINK-24356 > Project: Flink > Issue Type: Bug > Components: Build System / Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Julia Pedrycz >Assignee: Igal Shilman >Priority: Major > > My use case is processing delayed messages with ability to cancel them and > update their delay. Whole logic is to wait with sending messages from ingress > to egress. > At first I worked with embedded mode, with > org.apache.flink.statefun.sdk.Context. Just for testing I’ve made a function > that invoked cancelDelayedMessage() method and straight after sendAfter() > method. I used message id as cancellation token. My function worked just > fine, message was overridden (cancelled and send with new delay). > After that I wanted to switch to remote mode, so I used > org.apache.flink.statefun.sdk.java.Context. I’ve made similar function as > previously, but it did not work. Just to be sure that cancellation works, I > invoked sendAfter() and straight after that cancelDelayedMessage() method. > Messages were still appearing on egress topic after original delay. > Questions: > 1. Whether cancelDelayedMessage() is working in remote mode? > 2. Should it work the same like in embedded mode? > 3. Should it work as expected if we call cancelDelayedMessage() in a moment > after sendAfter()? (for testing purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24356) Not able to cancel delayed message using remote stateful function mode
[ https://issues.apache.org/jira/browse/FLINK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418558#comment-17418558 ] Igal Shilman commented on FLINK-24356: -- We will try to kick off the release process by the end of the month as we've already have some stuff accumulated. The PR for the fix will be open soon, is there a way for you to try it out? > Not able to cancel delayed message using remote stateful function mode > -- > > Key: FLINK-24356 > URL: https://issues.apache.org/jira/browse/FLINK-24356 > Project: Flink > Issue Type: Bug > Components: Build System / Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Julia Pedrycz >Assignee: Igal Shilman >Priority: Major > > My use case is processing delayed messages with ability to cancel them and > update their delay. Whole logic is to wait with sending messages from ingress > to egress. > At first I worked with embedded mode, with > org.apache.flink.statefun.sdk.Context. Just for testing I’ve made a function > that invoked cancelDelayedMessage() method and straight after sendAfter() > method. I used message id as cancellation token. My function worked just > fine, message was overridden (cancelled and send with new delay). > After that I wanted to switch to remote mode, so I used > org.apache.flink.statefun.sdk.java.Context. I’ve made similar function as > previously, but it did not work. Just to be sure that cancellation works, I > invoked sendAfter() and straight after that cancelDelayedMessage() method. > Messages were still appearing on egress topic after original delay. > Questions: > 1. Whether cancelDelayedMessage() is working in remote mode? > 2. Should it work the same like in embedded mode? > 3. Should it work as expected if we call cancelDelayedMessage() in a moment > after sendAfter()? (for testing purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-24356) Not able to cancel delayed message using remote stateful function mode
[ https://issues.apache.org/jira/browse/FLINK-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17418556#comment-17418556 ] Julia Pedrycz commented on FLINK-24356: --- Thank you for quick response. What is the ETA for the upcoming release? > Not able to cancel delayed message using remote stateful function mode > -- > > Key: FLINK-24356 > URL: https://issues.apache.org/jira/browse/FLINK-24356 > Project: Flink > Issue Type: Bug > Components: Build System / Stateful Functions >Affects Versions: statefun-3.1.0 >Reporter: Julia Pedrycz >Assignee: Igal Shilman >Priority: Major > > My use case is processing delayed messages with ability to cancel them and > update their delay. Whole logic is to wait with sending messages from ingress > to egress. > At first I worked with embedded mode, with > org.apache.flink.statefun.sdk.Context. Just for testing I’ve made a function > that invoked cancelDelayedMessage() method and straight after sendAfter() > method. I used message id as cancellation token. My function worked just > fine, message was overridden (cancelled and send with new delay). > After that I wanted to switch to remote mode, so I used > org.apache.flink.statefun.sdk.java.Context. I’ve made similar function as > previously, but it did not work. Just to be sure that cancellation works, I > invoked sendAfter() and straight after that cancelDelayedMessage() method. > Messages were still appearing on egress topic after original delay. > Questions: > 1. Whether cancelDelayedMessage() is working in remote mode? > 2. Should it work the same like in embedded mode? > 3. Should it work as expected if we call cancelDelayedMessage() in a moment > after sendAfter()? (for testing purposes) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dawidwys commented on pull request #17328: Group of hotfix commits fixing a couple of things in the tests
dawidwys commented on pull request #17328: URL: https://github.com/apache/flink/pull/17328#issuecomment-924857628 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17328: Group of hotfix commits fixing a couple of things in the tests
flinkbot edited a comment on pull request #17328: URL: https://github.com/apache/flink/pull/17328#issuecomment-924027879 ## CI report: * 1f918779ba37b1de5c5d6e08ee698b03f57de8dd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24389) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
flinkbot edited a comment on pull request #17331: URL: https://github.com/apache/flink/pull/17331#issuecomment-924142360 ## CI report: * 336d7f0aca4d20c49c1f6cb95ac3702511e8033c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24363) * 66b303dcd4c117307248cc2f80f2e50733b08b0e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24395) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
flinkbot edited a comment on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-921483643 ## CI report: * 8bab59ec4ddf8c0775e3199944be09f37aaa7a5d UNKNOWN * 1d20e8536f5cb590b904763fbcec1a5a2d074164 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24377) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24246) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24396) * 347481706d1752ab6cdef0ae207d0b286bf6daf2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17253: [FLINK-24182][task] Do not interrupt tasks/operators that are already closing
flinkbot edited a comment on pull request #17253: URL: https://github.com/apache/flink/pull/17253#issuecomment-917975072 ## CI report: * a9a487d8ce9a9112f67e42de61c7eb249bd2ea58 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24388) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on a change in pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
pnowojski commented on a change in pull request #17331:
URL: https://github.com/apache/flink/pull/17331#discussion_r713855871
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -1955,6 +1959,25 @@ private void abortPendingCheckpoint(
}
}
+private void handleCheckpointException(
+PendingCheckpoint pendingCheckpoint,
+CheckpointProperties checkpointProperties,
+CheckpointException exception,
+ExecutionAttemptID executionAttemptID) {
+if (checkpointProperties.isSavepoint()
+&& (checkpointProperties.isSynchronous() || pendingCheckpoint
== null)) {
+failureManager.handleSynchronousSavepointFailure(exception);
Review comment:
async phase of checkpoint has unfortunately nothing to do with
"synchronous savepoint". Synchronous savepoint is a bad internal name for a
savepoint barrier that is part of "stop-with-savepoint" feature. In this
context, non-synchronous savepoint are savepoints that are not followed by
stopping of the job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] akalash commented on a change in pull request #17331: [FLINK-24344][runtime] Calling the CheckpointFailureManager even if the exception happens during the checkpoint initialization
akalash commented on a change in pull request #17331:
URL: https://github.com/apache/flink/pull/17331#discussion_r713852975
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java
##
@@ -1955,6 +1959,25 @@ private void abortPendingCheckpoint(
}
}
+private void handleCheckpointException(
+PendingCheckpoint pendingCheckpoint,
+CheckpointProperties checkpointProperties,
+CheckpointException exception,
+ExecutionAttemptID executionAttemptID) {
+if (checkpointProperties.isSavepoint()
+&& (checkpointProperties.isSynchronous() || pendingCheckpoint
== null)) {
+failureManager.handleSynchronousSavepointFailure(exception);
Review comment:
My idea is if we fail during the initialization phase it means that
asynchronous part of the checkpoint/savepoint has not started yet so we can do
the same logic as for synchronous savepoint
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on pull request #17308: [FLINK-24291][table-planner]Decimal precision is lost when deserializing in test cases
xuyangzhong commented on pull request #17308: URL: https://github.com/apache/flink/pull/17308#issuecomment-924844336 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
