[GitHub] [flink] gaoyunhaii commented on a change in pull request #18805: [FLINK-26173][streaming] Recover GlobalCommittables with Sink V1 GlobalCommittable serializer
gaoyunhaii commented on a change in pull request #18805:
URL: https://github.com/apache/flink/pull/18805#discussion_r820063655
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/SinkWriterOperator.java
##
@@ -185,12 +219,36 @@ private void emitCommittables(Long checkpointId) throws
IOException, Interrupted
Collection committables =
((PrecommittingSinkWriter)
sinkWriter).prepareCommit();
StreamingRuntimeContext runtimeContext = getRuntimeContext();
-int indexOfThisSubtask = runtimeContext.getIndexOfThisSubtask();
+final int indexOfThisSubtask = runtimeContext.getIndexOfThisSubtask();
+final int numberOfParallelSubtasks =
runtimeContext.getNumberOfParallelSubtasks();
+
+// Emit only committable summary if there are legacy committables
+if (!legacyCommittables.isEmpty()) {
+emit(
Review comment:
Since it is called from `prepareSnapshotPreBarrier`, thus if the job is
restored from savepoint, the next checkpoint should start from at least the
`restored checkpoint id + 1`, thus it should always larger than
`InitContext.INITIAL_CHECKPOINT_ID` ?
The main issues of `checkpointId == InitContext.INITIAL_CHECKPOINT_ID` is
that in this case there will be two summary messages of the same checkpoint id,
which would cause exceptions in `CommitterOperator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18805: [FLINK-26173][streaming] Recover GlobalCommittables with Sink V1 GlobalCommittable serializer
gaoyunhaii commented on a change in pull request #18805: URL: https://github.com/apache/flink/pull/18805#discussion_r820063438 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java ## @@ -447,5 +451,13 @@ public void commit(Collection> committables) committables.forEach(CommitRequest::retryLater); Review comment: Thanks for the clarification! I did not notice the difference of types. Might we add some comments for this behavior? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on a change in pull request #18805: [FLINK-26173][streaming] Recover GlobalCommittables with Sink V1 GlobalCommittable serializer
gaoyunhaii commented on a change in pull request #18805: URL: https://github.com/apache/flink/pull/18805#discussion_r820063438 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java ## @@ -447,5 +451,13 @@ public void commit(Collection> committables) committables.forEach(CommitRequest::retryLater); Review comment: Thanks for the clarification! Might we add some comments for this~? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 6ea2dfde9fcd348318687821c84bdbd5164db99d Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32559) * d55c277b2822bf631c2cb22483c646ff2104d322 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32562) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32559) * d55c277b2822bf631c2cb22483c646ff2104d322 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18928: [FLINK-25545][flink-clients][JUnit5 Migration] Module: flink-clients.
flinkbot edited a comment on pull request #18928: URL: https://github.com/apache/flink/pull/18928#issuecomment-1054170977 ## CI report: * 667803d44d1d57ad3bdab11f66828dc5b959ca56 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32538) * 69a7198491e223208b865892c41a422fd86df5ed Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32561) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18928: [FLINK-25545][flink-clients][JUnit5 Migration] Module: flink-clients.
flinkbot edited a comment on pull request #18928: URL: https://github.com/apache/flink/pull/18928#issuecomment-1054170977 ## CI report: * 667803d44d1d57ad3bdab11f66828dc5b959ca56 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32538) * 69a7198491e223208b865892c41a422fd86df5ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * ee9e5abdd520ea150ebb8e6e82455e35ff10de28 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32558) * 7d987dc1ebefcd9e92ce6d85ceec313de745e227 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32560) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) * ee9e5abdd520ea150ebb8e6e82455e35ff10de28 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32558) * 7d987dc1ebefcd9e92ce6d85ceec313de745e227 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32559) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) * ee9e5abdd520ea150ebb8e6e82455e35ff10de28 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32558) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) * ee9e5abdd520ea150ebb8e6e82455e35ff10de28 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) * ee9e5abdd520ea150ebb8e6e82455e35ff10de28 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2959285b37c122e43011c0e6928f19535095ce07 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32537) * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) * ee9e5abdd520ea150ebb8e6e82455e35ff10de28 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * cbe0cd7606aed4937eaa68a9b806c7776487aaa4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32539) * 3f303f7d8cb071df7828f06d90818456e0fb94a7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32557) * 6ea2dfde9fcd348318687821c84bdbd5164db99d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * cbe0cd7606aed4937eaa68a9b806c7776487aaa4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32539) * 3f303f7d8cb071df7828f06d90818456e0fb94a7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2959285b37c122e43011c0e6928f19535095ce07 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32537) * 2b12c60dc4b735c3a358fe3f9b362253e674651a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32556) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17933: [FLINK-24948][jdbc] Fix bug that special character in column names breaks JDBC statement parsing
flinkbot edited a comment on pull request #17933: URL: https://github.com/apache/flink/pull/17933#issuecomment-980508183 ## CI report: * 2959285b37c122e43011c0e6928f19535095ce07 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32537) * 2b12c60dc4b735c3a358fe3f9b362253e674651a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MrWhiteSike commented on pull request #18655: [FLINK-25799] [docs] Translate table/filesystem.md page into Chinese.
MrWhiteSike commented on pull request #18655: URL: https://github.com/apache/flink/pull/18655#issuecomment-1059634434 [@wuchong](https://github.com/wuchong) Would you mind making the final check on it ? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-21364) piggyback finishedSplitIds in RequestSplitEvent
[
https://issues.apache.org/jira/browse/FLINK-21364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501576#comment-17501576
]
Steven Zhen Wu edited comment on FLINK-21364 at 3/4/22, 10:56 PM:
--
actually, I think we should look into FLIP-182 FLINK-18450 first and see if
Iceberg source can use the same mechanism to watermark alignment. cc [~sundaram]
was (Author: stevenz3wu):
actually, I think we should look into FLIP-182 FLINK-18450 first and see if
Iceberg source can use the same mechanism to watermark alignment.
> piggyback finishedSplitIds in RequestSplitEvent
> ---
>
> Key: FLINK-21364
> URL: https://issues.apache.org/jira/browse/FLINK-21364
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.12.1
>Reporter: Steven Zhen Wu
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> For some split assignment strategy, the enumerator/assigner needs to track
> the completed splits to advance watermark for event time alignment or rough
> ordering. Right now, `RequestSplitEvent` for FLIP-27 source doesn't support
> pass-along of the `finishedSplitIds` info and hence we have to create our own
> custom source event type for Iceberg source.
> Here is the proposal of add such optional info to `RequestSplitEvent`.
> {code}
> public RequestSplitEvent(
> @Nullable String hostName,
> @Nullable Collection finishedSplitIds)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-21364) piggyback finishedSplitIds in RequestSplitEvent
[
https://issues.apache.org/jira/browse/FLINK-21364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501576#comment-17501576
]
Steven Zhen Wu commented on FLINK-21364:
actually, I think we should look into FLIP-182 FLINK-18450 first and see if
Iceberg source can use the same mechanism to watermark alignment.
> piggyback finishedSplitIds in RequestSplitEvent
> ---
>
> Key: FLINK-21364
> URL: https://issues.apache.org/jira/browse/FLINK-21364
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.12.1
>Reporter: Steven Zhen Wu
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> For some split assignment strategy, the enumerator/assigner needs to track
> the completed splits to advance watermark for event time alignment or rough
> ordering. Right now, `RequestSplitEvent` for FLIP-27 source doesn't support
> pass-along of the `finishedSplitIds` info and hence we have to create our own
> custom source event type for Iceberg source.
> Here is the proposal of add such optional info to `RequestSplitEvent`.
> {code}
> public RequestSplitEvent(
> @Nullable String hostName,
> @Nullable Collection finishedSplitIds)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
flinkbot edited a comment on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1059437393 ## CI report: * dbb6ce5f77546baf2368fc2f23a30d956aa8b3a0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32552) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18980: [FLINK-26421] Use only EnvironmentSettings to configure the environment
flinkbot edited a comment on pull request #18980: URL: https://github.com/apache/flink/pull/18980#issuecomment-1059275933 ## CI report: * 91046dc10a250f5547cee5478c0b13f14d3903a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21364) piggyback finishedSplitIds in RequestSplitEvent
[
https://issues.apache.org/jira/browse/FLINK-21364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501528#comment-17501528
]
Steven Zhen Wu commented on FLINK-21364:
[~jqin] [~thw] [~pnowojski] I like to restart this discussion. Besides the
proposal in this PR, we can also go with the other approach that Thomas
mentioned where we create a separate `FinishedSplitsEvent`. Please let me know
your thoughts.
> piggyback finishedSplitIds in RequestSplitEvent
> ---
>
> Key: FLINK-21364
> URL: https://issues.apache.org/jira/browse/FLINK-21364
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.12.1
>Reporter: Steven Zhen Wu
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> For some split assignment strategy, the enumerator/assigner needs to track
> the completed splits to advance watermark for event time alignment or rough
> ordering. Right now, `RequestSplitEvent` for FLIP-27 source doesn't support
> pass-along of the `finishedSplitIds` info and hence we have to create our own
> custom source event type for Iceberg source.
> Here is the proposal of add such optional info to `RequestSplitEvent`.
> {code}
> public RequestSplitEvent(
> @Nullable String hostName,
> @Nullable Collection finishedSplitIds)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-21364) piggyback finishedSplitIds in RequestSplitEvent
[
https://issues.apache.org/jira/browse/FLINK-21364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501525#comment-17501525
]
Steven Zhen Wu commented on FLINK-21364:
[~pnowojski] sorry for missed you message earlier.
Yeah, the motivation is for watermark alignment for Iceberg source, where the
watermark alignment is happening in the enumerator. Hence the enumerator needs
to know which files/splits are completed and decides whether to advance
watermark or not.
https://github.com/stevenzwu/iceberg/blob/flip27IcebergSource/flink/v1.13/flink/src/main/java/org/apache/iceberg/flink/source/enumerator/AbstractIcebergEnumerator.java#L89
For sources with unbounded split (like Kafka), I believe the watermark
alignment is done at readers side.
> piggyback finishedSplitIds in RequestSplitEvent
> ---
>
> Key: FLINK-21364
> URL: https://issues.apache.org/jira/browse/FLINK-21364
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Affects Versions: 1.12.1
>Reporter: Steven Zhen Wu
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> For some split assignment strategy, the enumerator/assigner needs to track
> the completed splits to advance watermark for event time alignment or rough
> ordering. Right now, `RequestSplitEvent` for FLIP-27 source doesn't support
> pass-along of the `finishedSplitIds` info and hence we have to create our own
> custom source event type for Iceberg source.
> Here is the proposal of add such optional info to `RequestSplitEvent`.
> {code}
> public RequestSplitEvent(
> @Nullable String hostName,
> @Nullable Collection finishedSplitIds)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (FLINK-26487) Upgrade to Operator SDK 2.2.0 and replace custom informereventsource logic
[ https://issues.apache.org/jira/browse/FLINK-26487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora reassigned FLINK-26487: -- Assignee: Nicholas Jiang > Upgrade to Operator SDK 2.2.0 and replace custom informereventsource logic > -- > > Key: FLINK-26487 > URL: https://issues.apache.org/jira/browse/FLINK-26487 > Project: Flink > Issue Type: Sub-task >Reporter: Gyula Fora >Assignee: Nicholas Jiang >Priority: Major > > This ticket tracks the future required work to bump the operator sdk version > to 2.2.0 once it is released. > This release allows us to remove the custom InformerEventsource logic that we > added to the OperatorUtils class and use the built in functionality. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26487) Upgrade to Operator SDK 2.2.0 and replace custom informereventsource logic
[ https://issues.apache.org/jira/browse/FLINK-26487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501507#comment-17501507 ] Gyula Fora commented on FLINK-26487: of course! > Upgrade to Operator SDK 2.2.0 and replace custom informereventsource logic > -- > > Key: FLINK-26487 > URL: https://issues.apache.org/jira/browse/FLINK-26487 > Project: Flink > Issue Type: Sub-task >Reporter: Gyula Fora >Priority: Major > > This ticket tracks the future required work to bump the operator sdk version > to 2.2.0 once it is released. > This release allows us to remove the custom InformerEventsource logic that we > added to the OperatorUtils class and use the built in functionality. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18610: [FLINK-23843][runtime] Properly fail the job when SplitEnumeratorContext.runInCoordinatorThread() throws an exception
flinkbot edited a comment on pull request #18610: URL: https://github.com/apache/flink/pull/18610#issuecomment-1028011200 ## CI report: * 834d3cde88f2e8dd4d75654a7fc1770c042fd617 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32536) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26334) When timestamp - offset + windowSize < 0, elements cannot be assigned to the correct window
[
https://issues.apache.org/jira/browse/FLINK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501490#comment-17501490
]
realdengziqi commented on FLINK-26334:
--
[~Terry1897] Thank you, we will continue to work hard.
> When timestamp - offset + windowSize < 0, elements cannot be assigned to the
> correct window
> ---
>
> Key: FLINK-26334
> URL: https://issues.apache.org/jira/browse/FLINK-26334
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.15.0, 1.14.3
> Environment: flink version 1.14.3
>Reporter: realdengziqi
>Assignee: realdengziqi
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2022-03-04-11-28-26-616.png,
> image-2022-03-04-11-37-10-035.png
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> h2. issue
> Hello!
> When we were studying the flink source code, we found that there was
> a problem with its algorithm for calculating the window start time. When
> _timestamp - offset + windowSize < 0_ , the element will be incorrectly
> allocated to a window with a WindowSize larger than its own timestamp.
> The problem is in
> _org.apache.flink.streaming.api.windowing.windows.TimeWindow_
> {code:java}
> public static long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> return timestamp - (timestamp - offset + windowSize) % windowSize;
> } {code}
> _!image-2022-03-04-11-28-26-616.png|width=710,height=251!_
> We believe that this violates the constraints between time and
> window. *That is, an element should fall within a window whose start time is
> less than its own timestamp and whose end time is greater than its own
> timestamp.* However, the current situation is when {_}timestamp - offset +
> windowSize < 0{_}, *the element falls into a future time window.*
> *You can reproduce the bug with the code at the end of the post.*
> h2. Solution
> In fact, the original algorithm is no problem in python, the key to
> this problem is the processing of the remainder operation by the programming
> language.
> We finally think that it should be modified to the following
> algorithm.
> {code:java}
> public static long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> return timestamp
> - (timestamp - offset) % windowSize
> - (windowSize & (timestamp - offset) >> 63);
> } {code}
> _windowSize & (timestamp - offset) >> 63_ The function of this
> formula is to subtract windowSize from the overall operation result when
> {_}timestamp - offset<0{_}, otherwise do nothing. This way we can handle both
> positive and negative timestamps.
> Finally, the element can be assigned to the correct window.
> !image-2022-03-04-11-37-10-035.png|width=712,height=284!
> This code can pass current unit tests.
> h2. getWindowStartWithOffset methods in other packages
> I think that there should be many places in
> {_}getWindowStartWithOffset{_}. We searched for this method in the project
> and found that the problem of negative timestamps is handled in _flink.table._
> Below is their source code.
>
> _{{org.apache.flink.table.runtime.operators.window.grouping.WindowsGrouping}}_
> {code:java}
> private long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> long remainder = (timestamp - offset) % windowSize;
> // handle both positive and negative cases
> if (remainder < 0) {
> return timestamp - (remainder + windowSize);
> } else {
> return timestamp - remainder;
> }
> } {code}
> h2. Can we make a pull request?
> If the community deems it necessary to revise it, hopefully this task
> can be handed over to us. Our members are all students who have just
> graduated from school, and it is a great encouragement for us to contribute
> code to flink.
> Thank you so much!
> From Deng Ziqi & Lin Wanni & Guo Yuanfang
>
>
>
> h2. reproduce
> {code:java}
> /* output
> WindowStart: -15000ExactSize:1(a,-17000)
> WindowStart: -1ExactSize:1(b,-12000)
> WindowStart: -5000 ExactSize:2(c,-7000)
> WindowStart: -5000 ExactSize:2(d,-2000)
> WindowStart: 0 ExactSize:1(e,3000)
> WindowStart: 5000 ExactSize:1(f,8000)
> WindowStart: 1 ExactSize:1(g,13000)
> WindowStart: 15000 ExactSize:1(h,18000)
> */
> public class Example {
> public static void main(String[] args) throws Exception {
> final TimeZone timeZone = TimeZone.getTimeZone("GTM+0");
> TimeZone.setDefault(timeZone);
> StreamExecutionEnvironment env =
>
[jira] [Commented] (FLINK-26334) When timestamp - offset + windowSize < 0, elements cannot be assigned to the correct window
[
https://issues.apache.org/jira/browse/FLINK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501488#comment-17501488
]
realdengziqi commented on FLINK-26334:
--
[~martijnvisser] Thanks a lot, we just launched the PR :D
> When timestamp - offset + windowSize < 0, elements cannot be assigned to the
> correct window
> ---
>
> Key: FLINK-26334
> URL: https://issues.apache.org/jira/browse/FLINK-26334
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.15.0, 1.14.3
> Environment: flink version 1.14.3
>Reporter: realdengziqi
>Assignee: realdengziqi
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2022-03-04-11-28-26-616.png,
> image-2022-03-04-11-37-10-035.png
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> h2. issue
> Hello!
> When we were studying the flink source code, we found that there was
> a problem with its algorithm for calculating the window start time. When
> _timestamp - offset + windowSize < 0_ , the element will be incorrectly
> allocated to a window with a WindowSize larger than its own timestamp.
> The problem is in
> _org.apache.flink.streaming.api.windowing.windows.TimeWindow_
> {code:java}
> public static long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> return timestamp - (timestamp - offset + windowSize) % windowSize;
> } {code}
> _!image-2022-03-04-11-28-26-616.png|width=710,height=251!_
> We believe that this violates the constraints between time and
> window. *That is, an element should fall within a window whose start time is
> less than its own timestamp and whose end time is greater than its own
> timestamp.* However, the current situation is when {_}timestamp - offset +
> windowSize < 0{_}, *the element falls into a future time window.*
> *You can reproduce the bug with the code at the end of the post.*
> h2. Solution
> In fact, the original algorithm is no problem in python, the key to
> this problem is the processing of the remainder operation by the programming
> language.
> We finally think that it should be modified to the following
> algorithm.
> {code:java}
> public static long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> return timestamp
> - (timestamp - offset) % windowSize
> - (windowSize & (timestamp - offset) >> 63);
> } {code}
> _windowSize & (timestamp - offset) >> 63_ The function of this
> formula is to subtract windowSize from the overall operation result when
> {_}timestamp - offset<0{_}, otherwise do nothing. This way we can handle both
> positive and negative timestamps.
> Finally, the element can be assigned to the correct window.
> !image-2022-03-04-11-37-10-035.png|width=712,height=284!
> This code can pass current unit tests.
> h2. getWindowStartWithOffset methods in other packages
> I think that there should be many places in
> {_}getWindowStartWithOffset{_}. We searched for this method in the project
> and found that the problem of negative timestamps is handled in _flink.table._
> Below is their source code.
>
> _{{org.apache.flink.table.runtime.operators.window.grouping.WindowsGrouping}}_
> {code:java}
> private long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> long remainder = (timestamp - offset) % windowSize;
> // handle both positive and negative cases
> if (remainder < 0) {
> return timestamp - (remainder + windowSize);
> } else {
> return timestamp - remainder;
> }
> } {code}
> h2. Can we make a pull request?
> If the community deems it necessary to revise it, hopefully this task
> can be handed over to us. Our members are all students who have just
> graduated from school, and it is a great encouragement for us to contribute
> code to flink.
> Thank you so much!
> From Deng Ziqi & Lin Wanni & Guo Yuanfang
>
>
>
> h2. reproduce
> {code:java}
> /* output
> WindowStart: -15000ExactSize:1(a,-17000)
> WindowStart: -1ExactSize:1(b,-12000)
> WindowStart: -5000 ExactSize:2(c,-7000)
> WindowStart: -5000 ExactSize:2(d,-2000)
> WindowStart: 0 ExactSize:1(e,3000)
> WindowStart: 5000 ExactSize:1(f,8000)
> WindowStart: 1 ExactSize:1(g,13000)
> WindowStart: 15000 ExactSize:1(h,18000)
> */
> public class Example {
> public static void main(String[] args) throws Exception {
> final TimeZone timeZone = TimeZone.getTimeZone("GTM+0");
> TimeZone.setDefault(timeZone);
> StreamExecutionEnvironment env =
[jira] [Commented] (FLINK-26487) Upgrade to Operator SDK 2.2.0 and replace custom informereventsource logic
[ https://issues.apache.org/jira/browse/FLINK-26487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501486#comment-17501486 ] Nicholas Jiang commented on FLINK-26487: [~gyfora], could you please assign this ticket to me? I would like track the 2.2.0 release of the java-operator-sdk and do the working above mentioned. > Upgrade to Operator SDK 2.2.0 and replace custom informereventsource logic > -- > > Key: FLINK-26487 > URL: https://issues.apache.org/jira/browse/FLINK-26487 > Project: Flink > Issue Type: Sub-task >Reporter: Gyula Fora >Priority: Major > > This ticket tracks the future required work to bump the operator sdk version > to 2.2.0 once it is released. > This release allows us to remove the custom InformerEventsource logic that we > added to the OperatorUtils class and use the built in functionality. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
flinkbot edited a comment on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1059437393 ## CI report: * dbb6ce5f77546baf2368fc2f23a30d956aa8b3a0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32552) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
flinkbot commented on pull request #18982: URL: https://github.com/apache/flink/pull/18982#issuecomment-1059437393 ## CI report: * dbb6ce5f77546baf2368fc2f23a30d956aa8b3a0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26334) When timestamp - offset + windowSize < 0, elements cannot be assigned to the correct window
[
https://issues.apache.org/jira/browse/FLINK-26334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-26334:
---
Labels: pull-request-available (was: )
> When timestamp - offset + windowSize < 0, elements cannot be assigned to the
> correct window
> ---
>
> Key: FLINK-26334
> URL: https://issues.apache.org/jira/browse/FLINK-26334
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.15.0, 1.14.3
> Environment: flink version 1.14.3
>Reporter: realdengziqi
>Assignee: realdengziqi
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2022-03-04-11-28-26-616.png,
> image-2022-03-04-11-37-10-035.png
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> h2. issue
> Hello!
> When we were studying the flink source code, we found that there was
> a problem with its algorithm for calculating the window start time. When
> _timestamp - offset + windowSize < 0_ , the element will be incorrectly
> allocated to a window with a WindowSize larger than its own timestamp.
> The problem is in
> _org.apache.flink.streaming.api.windowing.windows.TimeWindow_
> {code:java}
> public static long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> return timestamp - (timestamp - offset + windowSize) % windowSize;
> } {code}
> _!image-2022-03-04-11-28-26-616.png|width=710,height=251!_
> We believe that this violates the constraints between time and
> window. *That is, an element should fall within a window whose start time is
> less than its own timestamp and whose end time is greater than its own
> timestamp.* However, the current situation is when {_}timestamp - offset +
> windowSize < 0{_}, *the element falls into a future time window.*
> *You can reproduce the bug with the code at the end of the post.*
> h2. Solution
> In fact, the original algorithm is no problem in python, the key to
> this problem is the processing of the remainder operation by the programming
> language.
> We finally think that it should be modified to the following
> algorithm.
> {code:java}
> public static long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> return timestamp
> - (timestamp - offset) % windowSize
> - (windowSize & (timestamp - offset) >> 63);
> } {code}
> _windowSize & (timestamp - offset) >> 63_ The function of this
> formula is to subtract windowSize from the overall operation result when
> {_}timestamp - offset<0{_}, otherwise do nothing. This way we can handle both
> positive and negative timestamps.
> Finally, the element can be assigned to the correct window.
> !image-2022-03-04-11-37-10-035.png|width=712,height=284!
> This code can pass current unit tests.
> h2. getWindowStartWithOffset methods in other packages
> I think that there should be many places in
> {_}getWindowStartWithOffset{_}. We searched for this method in the project
> and found that the problem of negative timestamps is handled in _flink.table._
> Below is their source code.
>
> _{{org.apache.flink.table.runtime.operators.window.grouping.WindowsGrouping}}_
> {code:java}
> private long getWindowStartWithOffset(long timestamp, long offset, long
> windowSize) {
> long remainder = (timestamp - offset) % windowSize;
> // handle both positive and negative cases
> if (remainder < 0) {
> return timestamp - (remainder + windowSize);
> } else {
> return timestamp - remainder;
> }
> } {code}
> h2. Can we make a pull request?
> If the community deems it necessary to revise it, hopefully this task
> can be handed over to us. Our members are all students who have just
> graduated from school, and it is a great encouragement for us to contribute
> code to flink.
> Thank you so much!
> From Deng Ziqi & Lin Wanni & Guo Yuanfang
>
>
>
> h2. reproduce
> {code:java}
> /* output
> WindowStart: -15000ExactSize:1(a,-17000)
> WindowStart: -1ExactSize:1(b,-12000)
> WindowStart: -5000 ExactSize:2(c,-7000)
> WindowStart: -5000 ExactSize:2(d,-2000)
> WindowStart: 0 ExactSize:1(e,3000)
> WindowStart: 5000 ExactSize:1(f,8000)
> WindowStart: 1 ExactSize:1(g,13000)
> WindowStart: 15000 ExactSize:1(h,18000)
> */
> public class Example {
> public static void main(String[] args) throws Exception {
> final TimeZone timeZone = TimeZone.getTimeZone("GTM+0");
> TimeZone.setDefault(timeZone);
> StreamExecutionEnvironment env =
>
[GitHub] [flink] realdengziqi opened a new pull request #18982: [FLINK-26334][datastream] Fix getWindowStartWithOffset in TimeWindow.java
realdengziqi opened a new pull request #18982:
URL: https://github.com/apache/flink/pull/18982
Co-authored-by: Lin WanNi
Co-authored-by: Guo YuanFang <1650213...@qq.com>
## What is the purpose of the change
The goal of this PR is to fix the bug that: the element couldn't be
assigned to the correct window-start, if it's *timestamp - offset + windowSize
< 0*.
This bug located at
_org.apache.flink.streaming.api.windowing.windows.TimeWindow_ .
This problem will be triggered by the negative timestamp, and is caused by
the calculation method of remainder in the JAVA compiler.
Specifically, when we try to calculate the window-start of an incoming
element, if _timestamp - offset + windowSize < 0_, based on the current
calculation formula for window-start, **the element will be right shifted to
the next window, which has a start time larger than the timestamp of current
element**, seems violated the assignment principle for elements on window.
This problem can be fixed by modifying the calculation formula inside the
getWindowStartWithOffset() method as below:
```java
public static long getWindowStartWithOffset(long timestamp, long offset,
long windowSize) {
return timestamp
- (timestamp - offset) % windowSize
- (windowSize & (timestamp - offset) >> 63);
}
```
After this modify, for the element who has negative timestamp, we can still
get the correct window-start. Like the below graph showing:
## Brief change log
- Fix getWindowStartWithOffset in *TimeWindow.java*
## Verifying this change
This change is already covered by existing tests, such as the tests in the
flink-streaming-java [mvn clean verify]
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18971: [FLINK-25969][table-runtime] Clean up caches in CompileUtils more aggressively
flinkbot edited a comment on pull request #18971: URL: https://github.com/apache/flink/pull/18971#issuecomment-1058052330 ## CI report: * 3e936f1361b87f455a89a65abcbcd6600139cf54 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32551) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18981: [FLINK-15550][runtime] Debug logging for TaskTest
flinkbot edited a comment on pull request #18981: URL: https://github.com/apache/flink/pull/18981#issuecomment-1059301393 ## CI report: * c28890d5370d21e096e08ee08f8c399675b8c990 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32550) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr closed pull request #18977: Update StreamingJobGraphGenerator.java
twalthr closed pull request #18977: URL: https://github.com/apache/flink/pull/18977 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #18977: Update StreamingJobGraphGenerator.java
twalthr commented on pull request #18977: URL: https://github.com/apache/flink/pull/18977#issuecomment-1059383653 This is not a typo: https://en.wikipedia.org/wiki/If_and_only_if -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18972: [FLINK-26453][clients] Prevent program configurations also when execu…
flinkbot edited a comment on pull request #18972: URL: https://github.com/apache/flink/pull/18972#issuecomment-1058144699 ## CI report: * e4cb01661266700467d17eb1a8ec8a5b26a6fc25 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32547) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18979: [FLINK-26484][fs] FileSystem.delete is not implemented consistently
flinkbot edited a comment on pull request #18979: URL: https://github.com/apache/flink/pull/18979#issuecomment-1059179598 ## CI report: * 07c30b4397a4242604b75f8ce9ab353c4672 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32548) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18975: [FLINK-26474][hive] fix the issue of failing to call some hive udf re…
flinkbot edited a comment on pull request #18975: URL: https://github.com/apache/flink/pull/18975#issuecomment-1058886429 ## CI report: * ab6650570c2b55e4e0b1cfa01e64f5039d20505c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32544) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26494) Missing logs during retry
[
https://issues.apache.org/jira/browse/FLINK-26494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-26494:
--
Fix Version/s: 1.15.0
> Missing logs during retry
> -
>
> Key: FLINK-26494
> URL: https://issues.apache.org/jira/browse/FLINK-26494
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Priority: Critical
> Fix For: 1.15.0
>
>
> The {{FutureRetry.retry}} functionality doesn't log the errors but just
> trigger a retry. This makes it harder for the user to figure out what's wrong.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26494) Missing logs during retry
Matthias Pohl created FLINK-26494:
-
Summary: Missing logs during retry
Key: FLINK-26494
URL: https://issues.apache.org/jira/browse/FLINK-26494
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Matthias Pohl
The {{FutureRetry.retry}} functionality doesn't log the errors but just trigger
a retry. This makes it harder for the user to figure out what's wrong.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] JingGe commented on pull request #18825: [FLINK-26126][kafka] fix numRecordsOut metric error
JingGe commented on pull request #18825: URL: https://github.com/apache/flink/pull/18825#issuecomment-1059346143 @fapaul would you please help merging this PR? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18825: [FLINK-26126][kafka] fix numRecordsOut metric error
flinkbot edited a comment on pull request #18825: URL: https://github.com/apache/flink/pull/18825#issuecomment-1043598537 ## CI report: * c9f2b3d7ff99d0116e28e2233eb9c91598a4ab52 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32546) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
flinkbot edited a comment on pull request #18746: URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702 ## CI report: * ec06532a2ea9edb00c82ce9304057966e5ef9ae7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18610: [FLINK-23843][runtime] Properly fail the job when SplitEnumeratorContext.runInCoordinatorThread() throws an exception
flinkbot edited a comment on pull request #18610: URL: https://github.com/apache/flink/pull/18610#issuecomment-1028011200 ## CI report: * 834d3cde88f2e8dd4d75654a7fc1770c042fd617 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32536) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dmvk commented on pull request #18610: [FLINK-23843][runtime] Properly fail the job when SplitEnumeratorContext.runInCoordinatorThread() throws an exception
dmvk commented on pull request #18610: URL: https://github.com/apache/flink/pull/18610#issuecomment-1059337610 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-26452) Flink deploy on k8s https SSLPeerUnverifiedException
[
https://issues.apache.org/jira/browse/FLINK-26452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
hjw closed FLINK-26452.
---
Release Note:
.kube/config add a line
insecure-skip-tls-verify : true
Resolution: Fixed
> Flink deploy on k8s https SSLPeerUnverifiedException
> -
>
> Key: FLINK-26452
> URL: https://issues.apache.org/jira/browse/FLINK-26452
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.13.6
>Reporter: hjw
>Priority: Major
>
> ~/.kube/config
> apiVersion:v1
> kind:config
> cluster:
> -name: "yf-dev-cluster1"
> cluster:
> server: "https://in-acpmanager.test.yfzx.cn/k8s/clusters/c-t5h2t;
> certificate-authority-data : “……"
> {code:java}
> 2022-03-02 18:59:30 | OkHttp
> https://in-acpmanager.test.yfzx.cn/...io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener
>
> Exec Failure javax.net.ssl.SSLPeerUnverifiedException Hostname
> in-acpmanager.test.yfzx.cn not verified:
> certificate: sha256/cw2T2s+Swhl7z+H35/3C1dTLxL26OOMO5VoEN9kAZCA=
> DN: CN=in-acpmanager.test.yfzx.cn
> subjectAltNames: []
> io.fabric8.kubernetes.client.KubernetesClientException: Failed to start
> websocket
> at
> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.onFailure(WatcherWebSocketListener.java:77)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:570)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket$1.onFailure(RealWebSocket.java:216)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.RealCall$AsyncCall.execute(RealCall.java:180)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Suppressed: java.lang.Throwable: waiting here
> at
> io.fabric8.kubernetes.client.utils.Utils.waitUntilReady(Utils.java:164)
> at
> io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:175)
> at
> io.fabric8.kubernetes.client.dsl.internal.WatcherWebSocketListener.waitUntilReady(WatcherWebSocketListener.java:120)
> at
> io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.waitUntilReady(WatchConnectionManager.java:82)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:705)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:678)
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.watch(BaseOperation.java:
> {code}
> {code:java}
> Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname
> in-acpmanager.test.yfzx.cn not verified:
> certificate: sha256/cw2T2s+Swhl7z+H35/3C1dTLxL26OOMO5VoEN9kAZCA=
> DN: CN=in-acpmanager.test.yfzx.cn
> subjectAltNames: []
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:350)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:300)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.RealConnection.connect(RealConnection.java:185)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
> at
> org.apache.flink.kubernetes.shaded.okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #18971: [FLINK-25969][table-runtime] Clean up caches in CompileUtils more aggressively
flinkbot edited a comment on pull request #18971: URL: https://github.com/apache/flink/pull/18971#issuecomment-1058052330 ## CI report: * 646d4fa75e140127098947f2702c10892e9686c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32500) * 3e936f1361b87f455a89a65abcbcd6600139cf54 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32551) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26473) Observer should support JobManager deployment crashed or deleted externally
[
https://issues.apache.org/jira/browse/FLINK-26473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501419#comment-17501419
]
Thomas Weise commented on FLINK-26473:
--
Yes, this appears to be a dup. I will have something out in a day or two.
> Observer should support JobManager deployment crashed or deleted externally
> ---
>
> Key: FLINK-26473
> URL: https://issues.apache.org/jira/browse/FLINK-26473
> Project: Flink
> Issue Type: Sub-task
> Components: Kubernetes Operator
>Reporter: Yang Wang
>Priority: Major
>
> Follow the discussion in this PR
> [https://github.com/apache/flink-kubernetes-operator/pull/26#discussion_r817514763.]
>
> Currently, the {{observeJmDeployment}} still could not cover some scenarios,
> e.g. JobManager deployment crashed, JobManager deployment was deleted
> externally. When it {{JobManagerDeploymentStatus}} comes to {{{}READY{}}}, it
> will always be {{{}READY{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18981: [FLINK-15550][runtime] Debug logging for TaskTest
flinkbot edited a comment on pull request #18981: URL: https://github.com/apache/flink/pull/18981#issuecomment-1059301393 ## CI report: * c28890d5370d21e096e08ee08f8c399675b8c990 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32550) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18971: [FLINK-25969][table-runtime] Clean up caches in CompileUtils more aggressively
flinkbot edited a comment on pull request #18971: URL: https://github.com/apache/flink/pull/18971#issuecomment-1058052330 ## CI report: * 646d4fa75e140127098947f2702c10892e9686c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32500) * 3e936f1361b87f455a89a65abcbcd6600139cf54 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18981: [FLINK-15550][runtime] Debug logging for TaskTest
flinkbot commented on pull request #18981: URL: https://github.com/apache/flink/pull/18981#issuecomment-1059301393 ## CI report: * c28890d5370d21e096e08ee08f8c399675b8c990 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26493) Use state machine based mechanism to simply the reconciler
[
https://issues.apache.org/jira/browse/FLINK-26493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501409#comment-17501409
]
Aitozi commented on FLINK-26493:
Agree, I will try to capture the current state transitions first.
> Use state machine based mechanism to simply the reconciler
> --
>
> Key: FLINK-26493
> URL: https://issues.apache.org/jira/browse/FLINK-26493
> Project: Flink
> Issue Type: Sub-task
>Reporter: Aitozi
>Priority: Major
>
> As discussed in
> [link|https://lists.apache.org/list?d...@flink.apache.org:lte=1M:controller%20flow]
> , we reach a consensus to use the state machine mechanism to simplify the
> annoying if-else in the reconciler. Since the modular {{Observer}}
> {{Reconciler}} and {{validator}} have completed. I think we can start this
> work now.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] akalash opened a new pull request #18981: [FLINK-15550][runtime] Debug logging for TaskTest
akalash opened a new pull request #18981: URL: https://github.com/apache/flink/pull/18981 ## What is the purpose of the change *Added extra debug logs for TaskTest* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #18979: [FLINK-26484][fs] FileSystem.delete is not implemented consistently
XComp commented on a change in pull request #18979:
URL: https://github.com/apache/flink/pull/18979#discussion_r819682581
##
File path: flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java
##
@@ -671,16 +672,111 @@ public boolean exists(final Path f) throws IOException {
}
/**
- * Delete a file.
+ * Deletes a file or directory.
*
- * @param f the path to delete
- * @param recursive if path is a directory and set to true,
the directory is
- * deleted else throws an exception. In case of a file the recursive
can be set to either
- * true or false
- * @return true if delete is successful, false
otherwise
- * @throws IOException
+ * @param path The path to the file that shall be deleted.
+ * @param recursive {@code true} will trigger a recursive deletion; {@code
false} will only try
+ * to delete the object referenced by the passed {@code path}. In the
latter case, a {@code
+ * DirectoryNotEmptyException} is thrown if the {@code path} refers to
a non-empty
+ * directory.
+ * @return {@code true} if the file or directory doesn't exist after this
call was processed.
+ * @throws java.nio.file.DirectoryNotEmptyException if the {@code path}
refers to a non-empty
+ * directory.
+ * @throws IOException If an error occurred while deleting the data
indicating that the deletion
+ * wasn't successful.
+ */
+public boolean delete(Path path, boolean recursive) throws IOException {
+if (recursive) {
+deleteRecursively(path);
+} else {
+deleteFile(path);
+}
+
+return true;
+}
+
+private void deleteRecursively(Path path) throws IOException {
+if (!exists(path)) {
+return;
+}
+
+final FileStatus[] containingFiles = listStatus(path);
+if (containingFiles == null) {
+LOG.warn(
+"No files could be retrieved even though the path was
marked as existing. This is an indication for a bug in the underlying
FileSystem implementation and should be reported. It won't affect the
processing of this run, though.");
+return;
+}
+
+IOException exception = null;
+for (FileStatus fileStatus : containingFiles) {
+final Path childPath = fileStatus.getPath();
+
+if (childPath.equals(path)) {
+// we need to make sure that we delete the path itself after
all its content is
+// deleted
+continue;
+}
+
+try {
+deleteRecursively(childPath);
+} catch (IOException e) {
+exception = ExceptionUtils.firstOrSuppressed(e, exception);
+}
+}
+
+if (exception == null) {
+deleteFile(path);
+} else {
+throw exception;
+}
+}
+
+/**
+ * Deletes the object referenced by the passed {@code path}.
+ *
+ * @param path The path referring to the object that shall be deleted.
+ * @throws java.nio.file.DirectoryNotEmptyException if the {@code path}
refers to a non-empty
+ * directory.
+ * @throws IOException if an error occurred while deleting the file other
than the {@code path}
+ * referring to a non-empty directory.
+ */
+private void deleteFile(Path path) throws IOException {
+if (isDirectoryWithContent(path)) {
+throw new DirectoryNotEmptyException(path.getPath());
+}
+
+try {
+tryToDelete(path);
+} catch (IOException e) {
+if (exists(path)) {
+throw e;
+}
+return;
+}
+
+if (exists(path)) {
+throw new IOException(path.getPath() + " could not be deleted for
unknown reasons.");
+}
+}
+
+/**
+ * Checks whether a given {@code path} refers to a non-empty directory.
+ *
+ * @param path The path referring to the directory in question.
+ * @return {@code true}, if the {@code path} refers to a non-empty
directory; otherwise {@code
+ * false}.
+ * @throws IOException If an error occurred while accessing the underlying
file system.
+ */
+protected abstract boolean isDirectoryWithContent(Path path) throws
IOException;
+
+/**
+ * Deletes the file or empty directory referenced by the given {@code
path}. Shouldn't do
+ * anything if the object doesn't exist.
+ *
+ * @param path The path to the object that is subject for deletion.
+ * @throws IOException If an error occurs while accessing the underlying
file system.
*/
-public abstract boolean delete(Path f, boolean recursive) throws
IOException;
Review comment:
Summary of an offline discussion: We might want to move this change
further down the class hierarchy to not affect the public interface
[GitHub] [flink] flinkbot edited a comment on pull request #18980: [FLINK-26421] Use only EnvironmentSettings to configure the environment
flinkbot edited a comment on pull request #18980: URL: https://github.com/apache/flink/pull/18980#issuecomment-1059275933 ## CI report: * 91046dc10a250f5547cee5478c0b13f14d3903a2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32549) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18957: [FLINK-26444][python]Window allocator supporting pyflink datastream API
flinkbot edited a comment on pull request #18957: URL: https://github.com/apache/flink/pull/18957#issuecomment-1056202149 ## CI report: * ec1e0a435186082e5ac1481bc093f9bdd9d94d70 UNKNOWN * 8e46affb05226eeed6b8eb20df971445139654a8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32535) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18979: [FLINK-26484][fs] FileSystem.delete is not implemented consistently
flinkbot edited a comment on pull request #18979: URL: https://github.com/apache/flink/pull/18979#issuecomment-1059179598 ## CI report: * ab2e26b71d343b548b87211b3a9ee6bf8ca4cf5a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32543) * 07c30b4397a4242604b75f8ce9ab353c4672 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32548) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18980: [FLINK-26421] Use only EnvironmentSettings to configure the environment
flinkbot commented on pull request #18980: URL: https://github.com/apache/flink/pull/18980#issuecomment-1059275933 ## CI report: * 91046dc10a250f5547cee5478c0b13f14d3903a2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26493) Use state machine based mechanism to simply the reconciler
[
https://issues.apache.org/jira/browse/FLINK-26493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501391#comment-17501391
]
Gyula Fora commented on FLINK-26493:
It would be good to first try to capture the current state transitions how the
operator works because it is not going to be the same as in the lyft operator.
Because most of our operations are synchronous, there are less states here I
think right now.
> Use state machine based mechanism to simply the reconciler
> --
>
> Key: FLINK-26493
> URL: https://issues.apache.org/jira/browse/FLINK-26493
> Project: Flink
> Issue Type: Sub-task
>Reporter: Aitozi
>Priority: Major
>
> As discussed in
> [link|https://lists.apache.org/list?d...@flink.apache.org:lte=1M:controller%20flow]
> , we reach a consensus to use the state machine mechanism to simplify the
> annoying if-else in the reconciler. Since the modular {{Observer}}
> {{Reconciler}} and {{validator}} have completed. I think we can start this
> work now.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-26476) Introduce flink-kubernetes-shaded to avoid overlapping classes
[
https://issues.apache.org/jira/browse/FLINK-26476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang closed FLINK-26476.
-
Resolution: Fixed
Fixed via:
main: 721a8680e395d64bbf0290c5faedeeb73353b0cc
> Introduce flink-kubernetes-shaded to avoid overlapping classes
> --
>
> Key: FLINK-26476
> URL: https://issues.apache.org/jira/browse/FLINK-26476
> Project: Flink
> Issue Type: Sub-task
> Components: Kubernetes Operator
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Labels: pull-request-available
>
> When we compile the flink-kubernetes-operator, we could find the following
> overlapping classes of fabric8 kubernetes client dependencies. I propose to
> add a flink-kubernetes-shaded module to fix this in current project. If we
> are confident enough the shading works well, I will contribute it to the
> upstream project Flink.
> {code:java}
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [WARNING] flink-kubernetes_2.12-1.14.3.jar,
> kubernetes-model-coordination-5.12.1.jar define 18 overlapping classes:
> [WARNING] -
> io.fabric8.kubernetes.api.model.coordination.v1.LeaseFluentImpl$SpecNestedImpl
> [WARNING] -
> io.fabric8.kubernetes.api.model.coordination.v1.LeaseListFluentImpl$ItemsNestedImpl
> [WARNING] - io.fabric8.kubernetes.api.model.coordination.v1.LeaseSpec
> [WARNING] -
> io.fabric8.kubernetes.api.model.coordination.v1.LeaseFluent$SpecNested
> [WARNING] -
> io.fabric8.kubernetes.api.model.coordination.v1.LeaseListFluentImpl
> [WARNING] - io.fabric8.kubernetes.api.model.coordination.v1.LeaseSpecBuilder
> [WARNING] - io.fabric8.kubernetes.api.model.coordination.v1.LeaseSpecFluent
> [WARNING] -
> io.fabric8.kubernetes.api.model.coordination.v1.LeaseFluentImpl$MetadataNestedImpl
> [WARNING] - io.fabric8.kubernetes.api.model.coordination.v1.LeaseFluent
> [WARNING] - io.fabric8.kubernetes.api.model.coordination.v1.LeaseListFluent
> [WARNING] - 8 more...
> [WARNING] flink-kubernetes_2.12-1.14.3.jar, kubernetes-client-5.12.1.jar
> define 476 overlapping classes:
> [WARNING] - io.fabric8.kubernetes.client.internal.CertUtils
> [WARNING] - io.fabric8.kubernetes.client.osgi.ManagedKubernetesClient
> [WARNING] - io.fabric8.kubernetes.client.CustomResource
> [WARNING] - io.fabric8.kubernetes.client.V1beta1ApiextensionAPIGroupDSL
> [WARNING] - io.fabric8.kubernetes.client.internal.PatchUtils$SingletonHolder
> [WARNING] - io.fabric8.kubernetes.client.VersionInfo$1
> [WARNING] - io.fabric8.kubernetes.client.utils.ReplaceValueStream
> [WARNING] - io.fabric8.kubernetes.client.dsl.ApiextensionsAPIGroupDSL
> [WARNING] - io.fabric8.kubernetes.client.dsl.CreateFromServerGettable
> [WARNING] - io.fabric8.kubernetes.client.dsl.Containerable
> [WARNING] - 466 more... {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-26493) Use state machine based mechanism to simply the reconciler
[
https://issues.apache.org/jira/browse/FLINK-26493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501381#comment-17501381
]
Aitozi commented on FLINK-26493:
I will try to define a initial state machine transfer flow we are target as
described in the dev mail
[https://github.com/lyft/flinkk8soperator/blob/master/docs/state_machine.md]
> Use state machine based mechanism to simply the reconciler
> --
>
> Key: FLINK-26493
> URL: https://issues.apache.org/jira/browse/FLINK-26493
> Project: Flink
> Issue Type: Sub-task
>Reporter: Aitozi
>Priority: Major
>
> As discussed in
> [link|https://lists.apache.org/list?d...@flink.apache.org:lte=1M:controller%20flow]
> , we reach a consensus to use the state machine mechanism to simplify the
> annoying if-else in the reconciler. Since the modular {{Observer}}
> {{Reconciler}} and {{validator}} have completed. I think we can start this
> work now.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-26493) Use state machine based mechanism to simply the reconciler
[
https://issues.apache.org/jira/browse/FLINK-26493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501379#comment-17501379
]
Aitozi edited comment on FLINK-26493 at 3/4/22, 3:22 PM:
-
Do you have some inputs for this [~gyfora] [~wangyang0918] [~t...@apache.org] ?
was (Author: aitozi):
Do you have some inputs for this [~gyfora] [~wangyang0918] ?
> Use state machine based mechanism to simply the reconciler
> --
>
> Key: FLINK-26493
> URL: https://issues.apache.org/jira/browse/FLINK-26493
> Project: Flink
> Issue Type: Sub-task
>Reporter: Aitozi
>Priority: Major
>
> As discussed in
> [link|https://lists.apache.org/list?d...@flink.apache.org:lte=1M:controller%20flow]
> , we reach a consensus to use the state machine mechanism to simplify the
> annoying if-else in the reconciler. Since the modular {{Observer}}
> {{Reconciler}} and {{validator}} have completed. I think we can start this
> work now.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-table-store] LadyForest commented on a change in pull request #28: [FLINK-26423] Integrate log store to StoreSink
LadyForest commented on a change in pull request #28:
URL: https://github.com/apache/flink-table-store/pull/28#discussion_r819662125
##
File path:
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/sink/StoreSink.java
##
@@ -123,22 +191,20 @@ public StoreSink(
});
}
-return new StoreGlobalCommitter<>(
-commit, fileStore.newExpire(), fileCommitSerializer(), lock,
overwritePartition);
+return new StoreGlobalCommitter(commit, fileStore.newExpire(), lock,
overwritePartition);
}
+@SuppressWarnings("unchecked")
@Override
public SimpleVersionedSerializer getCommittableSerializer() {
-return CommittableSerializer.INSTANCE;
+return new CommittableSerializer(
+fileCommitSerializer(), (SimpleVersionedSerializer)
logCommitSerializer());
}
@Override
-public GlobalCommittableSerializer
getGlobalCommittableSerializer() {
-ManifestCommittableSerializer fileCommSerializer =
-new ManifestCommittableSerializer(
-project(rowType, partitions), project(rowType, keys),
rowType);
-SimpleVersionedSerializer logCommitSerializer = new
NoOutputSerializer<>();
-return new GlobalCommittableSerializer<>(logCommitSerializer,
fileCommSerializer);
+public ManifestCommittableSerializer getGlobalCommittableSerializer() {
+return new ManifestCommittableSerializer(
+project(rowType, partitions), project(rowType, keys), rowType);
Review comment:
`new ManifestCommittableSerializer(project(rowType, partitions),
project(rowType, keys), rowType)` can be replaced with `new
ManifestCommittableSerializer(partitionType, KeyType, ValueType)` now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-26493) Use state machine based mechanism to simply the reconciler
[
https://issues.apache.org/jira/browse/FLINK-26493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17501379#comment-17501379
]
Aitozi commented on FLINK-26493:
Do you have some inputs for this [~gyfora] [~wangyang0918] ?
> Use state machine based mechanism to simply the reconciler
> --
>
> Key: FLINK-26493
> URL: https://issues.apache.org/jira/browse/FLINK-26493
> Project: Flink
> Issue Type: Sub-task
>Reporter: Aitozi
>Priority: Major
>
> As discussed in
> [link|https://lists.apache.org/list?d...@flink.apache.org:lte=1M:controller%20flow]
> , we reach a consensus to use the state machine mechanism to simplify the
> annoying if-else in the reconciler. Since the modular {{Observer}}
> {{Reconciler}} and {{validator}} have completed. I think we can start this
> work now.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-26493) Use state machine based mechanism to simply the reconciler
Aitozi created FLINK-26493:
--
Summary: Use state machine based mechanism to simply the reconciler
Key: FLINK-26493
URL: https://issues.apache.org/jira/browse/FLINK-26493
Project: Flink
Issue Type: Sub-task
Reporter: Aitozi
As discussed in
[link|https://lists.apache.org/list?d...@flink.apache.org:lte=1M:controller%20flow]
, we reach a consensus to use the state machine mechanism to simplify the
annoying if-else in the reconciler. Since the modular {{Observer}}
{{Reconciler}} and {{validator}} have completed. I think we can start this work
now.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-26421) Cleanup EnvironmentSettings and only use ConfigOptions
[ https://issues.apache.org/jira/browse/FLINK-26421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26421: --- Labels: pull-request-available (was: ) > Cleanup EnvironmentSettings and only use ConfigOptions > -- > > Key: FLINK-26421 > URL: https://issues.apache.org/jira/browse/FLINK-26421 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Marios Trivyzas >Assignee: Marios Trivyzas >Priority: Major > Labels: pull-request-available > > * integrate Configuration into EnvironmentSettings > * EnvironmentSettings should only contain a Configuration, not other members > (create config options for all members) > * Remove `TableEnvironmentImpl.create(EnvSettings, Configuration)` -> > create(EnvSettings) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] matriv opened a new pull request #18980: [FLINK-26421] Use only EnvironmentSettings to configure the environment
matriv opened a new pull request #18980: URL: https://github.com/apache/flink/pull/18980 ## What is the purpose of the change Use `EnvironmentSettings` with the new method `withConfiguration` in its `Builder` to specify configuration options on top of the one inherited by the environment (flink-conf.yml, CLI params). ## Brief change log - Remove planner & executor string identifiers - Remove deprecated create methods which use `TableConfig` instead of `EnvironmentSettings`. - Add an underlying `Configuration` in `EnvironmentSettings` so that they contain any extra use defined config options. - The `TableConfig` now implements `ReadableConfig and contains all of the config options specified by the env (`flink-conf.yml`, `CLI` params), in the underlying `rootConfiguration`, plus all the extra configuration defined by the user app through the `EnvironmentSettings`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): **no** - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **yes** - If yes, how is the feature documented? **docs / JavaDocs** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18979: [FLINK-26484][fs] FileSystem.delete is not implemented consistently
flinkbot edited a comment on pull request #18979: URL: https://github.com/apache/flink/pull/18979#issuecomment-1059179598 ## CI report: * ab2e26b71d343b548b87211b3a9ee6bf8ca4cf5a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32543) * 07c30b4397a4242604b75f8ce9ab353c4672 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18979: [FLINK-26484][fs] FileSystem.delete is not implemented consistently
flinkbot edited a comment on pull request #18979: URL: https://github.com/apache/flink/pull/18979#issuecomment-1059179598 ## CI report: * ab2e26b71d343b548b87211b3a9ee6bf8ca4cf5a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32543) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18978: [FLINK-25549][flink-dstl] [JUnit5 Migration] Module: flink-dstl
flinkbot edited a comment on pull request #18978: URL: https://github.com/apache/flink/pull/18978#issuecomment-1059111709 ## CI report: * cbe0cd7606aed4937eaa68a9b806c7776487aaa4 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32539) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18972: [FLINK-26453][clients] Prevent program configurations also when execu…
flinkbot edited a comment on pull request #18972: URL: https://github.com/apache/flink/pull/18972#issuecomment-1058144699 ## CI report: * 41dc5388514d8d2b8b76b3e752214be0134ffa62 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32505) * e4cb01661266700467d17eb1a8ec8a5b26a6fc25 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32547) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18928: [FLINK-25545][flink-clients][JUnit5 Migration] Module: flink-clients.
flinkbot edited a comment on pull request #18928: URL: https://github.com/apache/flink/pull/18928#issuecomment-1054170977 ## CI report: * 667803d44d1d57ad3bdab11f66828dc5b959ca56 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32538) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18825: [FLINK-26126][kafka] fix numRecordsOut metric error
flinkbot edited a comment on pull request #18825: URL: https://github.com/apache/flink/pull/18825#issuecomment-1043598537 ## CI report: * 9cb6c81fe9187633b2de072d0d1006b1dd3a1385 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32541) * c9f2b3d7ff99d0116e28e2233eb9c91598a4ab52 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32546) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18746: [FLINK-26162][docs]revamp security pages
flinkbot edited a comment on pull request #18746: URL: https://github.com/apache/flink/pull/18746#issuecomment-1038829702 ## CI report: * bc1fbfac29ef3d3ec51ecebf2ea3fa155148bfe6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32542) * ec06532a2ea9edb00c82ce9304057966e5ef9ae7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=32545) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fapaul commented on pull request #18805: [FLINK-26173][streaming] Recover GlobalCommittables with Sink V1 GlobalCommittable serializer
fapaul commented on pull request #18805: URL: https://github.com/apache/flink/pull/18805#issuecomment-1059227883 > It seems currently `filterRecoveredCommittables` and `combine` are used only on the legacy state~? For example, suppose we recovered from sink v1, then on restoring we would call `filterRecoveredCommittables`, then if the job continue running and take another several checkpoints, then after failover all committables are stored in the `CommittableCollector` and on restoring `filterRecoveredCommittables` seems would not be called~? I thought about this for a longer time and think the behaviour of `combine` should be fine since it is always used when calling commit. The semantic of `filterRecoveredCommittables` is definitely currently unclear although I would also question the purpose of this method. In general, I think all commit calls be for the committer of global committer need to be idempotent and handling already committed committables needs to happen in the commit method either way. I do not see an immediate benefit of calling `filterRecoveredCommittables`. I am not really what to do about that. Do you have an idea? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fapaul commented on a change in pull request #18805: [FLINK-26173][streaming] Recover GlobalCommittables with Sink V1 GlobalCommittable serializer
fapaul commented on a change in pull request #18805: URL: https://github.com/apache/flink/pull/18805#discussion_r819633484 ## File path: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java ## @@ -447,5 +451,13 @@ public void commit(Collection> committables) committables.forEach(CommitRequest::retryLater); Review comment: This is not easily possible because only committable are retriable at this point. I would need to add a special logic when this happens. I rely here on the fact that I assume all `commit` calls are idempotent because it might happen that global committables are committed again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
