[GitHub] [flink-table-store] zjureel commented on pull request #296: [FLINK-29345] Create reusing reader/writer config in orc format
zjureel commented on PR #296: URL: https://github.com/apache/flink-table-store/pull/296#issuecomment-1251875743 Hi @JingsongLi Could you help to review this pr when you're free, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29345) Too many open files in table store orc writer
[ https://issues.apache.org/jira/browse/FLINK-29345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29345: --- Labels: pull-request-available (was: ) > Too many open files in table store orc writer > - > > Key: FLINK-29345 > URL: https://issues.apache.org/jira/browse/FLINK-29345 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.3.0 > > Attachments: image-2022-09-20-11-57-11-373.png > > > !image-2022-09-20-11-57-11-373.png! > We can avoid reading the local file to obtain the config every time we create > a new writer by reusing the prepared configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] zjureel opened a new pull request, #296: [FLINK-29345] Create reusing reader/writer config in orc format
zjureel opened a new pull request, #296: URL: https://github.com/apache/flink-table-store/pull/296 Currently `OrcFileFormat` will create new `org.apache.hadoop.conf.Configuration` instance in methods `createReaderFactory` and `createWriterFactory`. The initialization of `org.apache.hadoop.conf.Configuration` tries to load local file `core-site.xml`. This PR will initialize the `writerConf` and `readerConf` of `org.apache.hadoop.conf.Configuration` and reuse them in `createReaderFactory` and `createWriterFactory`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub opened a new pull request, #157: [FLINK-28906] Support windowing in AgglomerativeClustering
yunfengzhou-hub opened a new pull request, #157: URL: https://github.com/apache/flink-ml/pull/157 ## What is the purpose of the change This PR introduces `Window` interface and its several implementations that can be used to slice input stream into finite batches and fed into Flink ML stages. On the basis of these changes, This PR adds a window parameter to AgglomerativeClustering so that it can work on unbounded input stream. ## Brief change log - Add `Window` interface and its subclasses, `BoundedWindow`, `SessionWindow` and `TumbleWindow`. - Add window parameter on AgglomerativeClustering and make it able to deal with infinite input stream. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with @public(Evolving): (no) - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (Java doc) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29345) Too many open files in table store orc writer
[ https://issues.apache.org/jira/browse/FLINK-29345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606916#comment-17606916 ] Shammon commented on FLINK-29345: - Hi [~lzljs3620320] I'd like to pick this issue to start my contribution to flink-table-store, can you assign it to me? THX > Too many open files in table store orc writer > - > > Key: FLINK-29345 > URL: https://issues.apache.org/jira/browse/FLINK-29345 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.3.0 > > Attachments: image-2022-09-20-11-57-11-373.png > > > !image-2022-09-20-11-57-11-373.png! > We can avoid reading the local file to obtain the config every time we create > a new writer by reusing the prepared configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the places I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a `CheckpointProperty` will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured `CheckpointProperty` was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. Would it be better just to start with our specific use case - always only
trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30),
and then allow community to iterate on this API?
2. Or we already know quite a few other use cases for checkpoint manual
triggering so we can start with a more general API? In this case, what are the
options we should expose in this API? Or maybe better to start a mailing list
thread / FLIP to discuss this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the places I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a `CheckpointProperty` will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured `CheckpointProperty` was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. Would it be better just to start with our specific use case - always only
trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30),
and then allow community to iterate on this API?
2. Or we already know quite a few other use cases for checkpoint manual
triggering so we can start with a more general API? In this case, what are the
options we should expose in API? Or maybe better to start a mailing list thread
/ FLIP to discuss this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the places I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a `CheckpointProperty` will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured `CheckpointProperty` was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. Would it be better just to start with our specific use case - always only
trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30),
and then allow community to iterate on this?
2. Or we already know quite a few other use cases for checkpoint manual
triggering so we can start with a more general API? In this case, what are the
options we should expose in API? Or maybe better to start a mailing list thread
/ FLIP to discuss this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the places I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a `CheckpointProperty` will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured `CheckpointProperty` was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. What checkpoint configurations we should expose in the restful API? Are
there more general use cases requesting other configurations for checkpoint
manual triggering (other than ours)? Or maybe better to start a mailing list
thread / FLIP to discuss this?
2. Or would it be better just to start with our specific use case - always
only trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the places I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a `CheckpointProperty` will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured CheckpointProperty was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. What checkpoint configurations we should expose in the restful API? Are
there more general use cases requesting other configurations for checkpoint
manual triggering (other than ours)?
2. Or would it be better just to start with our specific use case - always
only trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30)
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the places I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a `CheckpointProperty` will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured `CheckpointProperty` was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. What checkpoint configurations we should expose in the restful API? Are
there more general use cases requesting other configurations for checkpoint
manual triggering (other than ours)?
2. Or would it be better just to start with our specific use case - always
only trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the places I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a CheckpointProperty will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured CheckpointProperty was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. What checkpoint configurations we should expose in the restful API? Are
there more general use cases requesting other configurations for checkpoint
manual triggering (other than ours)?
2. Or would it be better just to start with our specific use case - always
only trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the place I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a CheckpointProperty will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured CheckpointProperty was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`. And I did not notice the
`@Internal` for the `CheckpointRetentionPolicy` just saw it was `public`.
Meanwhile I would love to hear your thoughts
1. What checkpoint configurations we should expose in the restful API? Are
there more general use cases requesting other configurations for checkpoint
manual triggering (other than ours)?
2. Or would it be better just to start with our specific use case - always
only trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the place I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a CheckpointProperty will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones, so I was thinking of creating one as close to an existing
pre-configured as possible. Meanwhile the public method to create a
pre-configured CheckpointProperty was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`.
Meanwhile I would love to hear your thoughts
1. What checkpoint configurations we should expose in the restful API? Are
there more general use cases requesting other configurations for checkpoint
manual triggering (other than ours)?
2. Or would it be better just to start with our specific use case - always
only trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r974889748
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointRetentionPolicy.java:
##
@@ -24,6 +24,9 @@
@Internal
public enum CheckpointRetentionPolicy {
+/** Full Checkpoints should be retained on cancellation and failure. */
+FULL_RETAIN_ON_CANCELLATION,
Review Comment:
Good catch @pnowojski. This was one of the place I was not very confident
with. As a hindsight, totally agree it was a bad design.
The reason why I did that was as following:
I was not sure if a CheckpointProperty will be valid if it is too far away
from
[these](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L246)
pre-configured ones so I was thinking of creating one as close to an existing
one as possible. Meanwhile the public method to create a pre-configured
CheckpointProperty was through
[this](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointProperties.java#L349)
method with a `CheckpointRetentionPolicy`.
Meanwhile I would love to hear your thoughts
1. What checkpoint configurations we should expose in the restful API? Are
there more general use cases requesting other configurations for checkpoint
manual triggering (other than ours)?
2. Or would it be better just to start with our specific use case - always
only trigger a full checkpoint, i.e. reuse all `CheckpointProperty` and only
replace `CheckpointType` with [`FULL_CHECKPOINT
`](https://github.com/apache/flink/blob/d8a4304b892412eab4c5c19b5deb84166943d3bb/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointType.java#L30)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20859: [release] release notes for the 1.16 release
flinkbot commented on PR #20859: URL: https://github.com/apache/flink/pull/20859#issuecomment-1251836872 ## CI report: * 7d53d0e56055d03bd5c3a57c1f4fb5eb8cb9e7b5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on pull request #20849: [FLINK-29325][documentation]Fix documentation bug on how to enable ba…
Myasuka commented on PR #20849: URL: https://github.com/apache/flink/pull/20849#issuecomment-1251828621 The CI on [a forked pipeline](https://dev.azure.com/myasuka/flink/_build/results?buildId=435=results) could pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29315) HDFSTest#testBlobServerRecovery fails on CI
[
https://issues.apache.org/jira/browse/FLINK-29315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606878#comment-17606878
]
Yun Tang commented on FLINK-29315:
--
It seems only apache-flink CI pipeline could fail, and my pipeline could pass:
[https://dev.azure.com/myasuka/flink/_build/results?buildId=435=results]
> HDFSTest#testBlobServerRecovery fails on CI
> ---
>
> Key: FLINK-29315
> URL: https://issues.apache.org/jira/browse/FLINK-29315
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.16.0, 1.15.2
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.16.0, 1.17.0
>
>
> The test started failing 2 days ago on different branches. I suspect
> something's wrong with the CI infrastructure.
> {code:java}
> Sep 15 09:11:22 [ERROR] Failures:
> Sep 15 09:11:22 [ERROR] HDFSTest.testBlobServerRecovery Multiple Failures
> (2 failures)
> Sep 15 09:11:22 java.lang.AssertionError: Test failed Error while
> running command to get file permissions : java.io.IOException: Cannot run
> program "ls": error=1, Operation not permitted
> Sep 15 09:11:22 at
> java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:913)
> Sep 15 09:11:22 at org.apache.hadoop.util.Shell.run(Shell.java:869)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1264)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1246)
> Sep 15 09:11:22 at
> org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1089)
> Sep 15 09:11:22 at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:697)
> Sep 15 09:11:22 at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:672)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:233)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:141)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2580)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2622)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2604)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2497)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1501)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:851)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:485)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
> Sep 15 09:11:22 at
> org.apache.flink.hdfstests.HDFSTest.createHDFS(HDFSTest.java:93)
> Sep 15 09:11:22 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Sep 15 09:11:22 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Sep 15 09:11:22 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Sep 15 09:11:22 at
> java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> Sep 15 09:11:22 ... 67 more
> Sep 15 09:11:22
> Sep 15 09:11:22 java.lang.NullPointerException:
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] leletan commented on pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on PR #20852: URL: https://github.com/apache/flink/pull/20852#issuecomment-1251824681 Also I am wondering if this change would need any further test coverages? If yes, where would be the best place I should start looking? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852: URL: https://github.com/apache/flink/pull/20852#discussion_r974870165 ## flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMasterGateway.java: ## @@ -218,13 +220,27 @@ CompletableFuture triggerSavepoint( final SavepointFormatType formatType, @RpcTimeout final Time timeout); +/** + * Triggers taking a checkpoint of the executed job. + * + * @param checkpointProperties to determine how checkpoint should be taken or null if the + * existing checkpoint property should be used + * @param timeout for the rpc call + * @return Future which is completed with the savepoint path once completed + */ +CompletableFuture triggerCheckpoint( Review Comment: Good question, @pnowojski. Assuming we are talking about 1. `CompletableFuture triggerCheckpoint(@RpcTimeout final Time timeout)` as the other method 2. `CompletedOperationCache checkpointTriggerCache` in the `DispatcherCachedOperationsHandler` as the cache (location as cache value?) We definitely can. Meanwhile I guess this means that the restful API (checkpoint trigger status API) will eventually return checkpoint location in the restful response body, instead of checkpoint id. The reason for returning `CompletedCheckpoint` was to facilitate both old method and new API: the new restful API was returning checkpoint id, while the other method was returning checkpoint location, and CompletedCheckpoint contains both. I am pretty new to flink so I am not sure what people typically care about a checkpoint. If we think it is a better design to return checkpoint location, instead of checkpoint id in the response body in the restful API, I am happy to return checkpoint location and reuse existing cache and another method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852: URL: https://github.com/apache/flink/pull/20852#discussion_r974874914 ## flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMasterGateway.java: ## @@ -218,13 +220,27 @@ CompletableFuture triggerSavepoint( final SavepointFormatType formatType, @RpcTimeout final Time timeout); +/** + * Triggers taking a checkpoint of the executed job. + * + * @param checkpointProperties to determine how checkpoint should be taken or null if the + * existing checkpoint property should be used + * @param timeout for the rpc call + * @return Future which is completed with the savepoint path once completed Review Comment: Good catch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leletan commented on pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on PR #20852: URL: https://github.com/apache/flink/pull/20852#issuecomment-1251821677 > Thanks a lot for the contribution @leletan . It looks mostly good to me, I've left a couple of comments. It would be great if someone else (maybe @zentol ) could also take a look at the REST API changes. > > Also there seems to be a build failure. > > > If yes, how is the feature documented? [WIP] > > Let's not forget about this :) Thanks for the super fast response, @pnowojski ! Do we have a standard convention/process about restful API feature? - The only thing I found so far is [this](https://github.com/apache/flink/tree/master/flink-docs#rest-api-documentation) README. I am still a little bit confused after reading the README. Do we have an example for that? - Any restful API documentation other than above we need to add? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29315) HDFSTest#testBlobServerRecovery fails on CI
[
https://issues.apache.org/jira/browse/FLINK-29315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606871#comment-17606871
]
Yang Wang commented on FLINK-29315:
---
[~mapohl] Sorry for the late response. I will help with [~hxb] to figure out
the problem of the Alibaba CI machines.
> HDFSTest#testBlobServerRecovery fails on CI
> ---
>
> Key: FLINK-29315
> URL: https://issues.apache.org/jira/browse/FLINK-29315
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / FileSystem, Tests
>Affects Versions: 1.16.0, 1.15.2
>Reporter: Chesnay Schepler
>Priority: Blocker
> Fix For: 1.16.0, 1.17.0
>
>
> The test started failing 2 days ago on different branches. I suspect
> something's wrong with the CI infrastructure.
> {code:java}
> Sep 15 09:11:22 [ERROR] Failures:
> Sep 15 09:11:22 [ERROR] HDFSTest.testBlobServerRecovery Multiple Failures
> (2 failures)
> Sep 15 09:11:22 java.lang.AssertionError: Test failed Error while
> running command to get file permissions : java.io.IOException: Cannot run
> program "ls": error=1, Operation not permitted
> Sep 15 09:11:22 at
> java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell.runCommand(Shell.java:913)
> Sep 15 09:11:22 at org.apache.hadoop.util.Shell.run(Shell.java:869)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1264)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.Shell.execCommand(Shell.java:1246)
> Sep 15 09:11:22 at
> org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1089)
> Sep 15 09:11:22 at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:697)
> Sep 15 09:11:22 at
> org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:672)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:233)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.DiskChecker.checkDirInternal(DiskChecker.java:141)
> Sep 15 09:11:22 at
> org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:116)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode$DataNodeDiskChecker.checkDir(DataNode.java:2580)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.checkStorageLocations(DataNode.java:2622)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2604)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2497)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1501)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:851)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:485)
> Sep 15 09:11:22 at
> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:444)
> Sep 15 09:11:22 at
> org.apache.flink.hdfstests.HDFSTest.createHDFS(HDFSTest.java:93)
> Sep 15 09:11:22 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Sep 15 09:11:22 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Sep 15 09:11:22 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Sep 15 09:11:22 at
> java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
> Sep 15 09:11:22 ... 67 more
> Sep 15 09:11:22
> Sep 15 09:11:22 java.lang.NullPointerException:
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29346) File naming configuration for filesink output
Kwang-in (Dennis) JUNG created FLINK-29346: -- Summary: File naming configuration for filesink output Key: FLINK-29346 URL: https://issues.apache.org/jira/browse/FLINK-29346 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem Affects Versions: 1.13.6 Reporter: Kwang-in (Dennis) JUNG Hello, I've made some research about file sink, but I couldn't find configuration about file naming (based on elements at stream). https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/file_sink/ Are there some reason about this? Because I need naming option of outputs, based on data in stream. Thanks. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27101) Periodically break the chain of incremental checkpoint
[ https://issues.apache.org/jira/browse/FLINK-27101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27101: --- Labels: pull-request-available (was: ) > Periodically break the chain of incremental checkpoint > -- > > Key: FLINK-27101 > URL: https://issues.apache.org/jira/browse/FLINK-27101 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing >Reporter: Steven Zhen Wu >Assignee: Jiale Tan >Priority: Major > Labels: pull-request-available > > Incremental checkpoint is almost a must for large-state jobs. It greatly > reduces the bytes uploaded to DFS per checkpoint. However, there are a few > implications from incremental checkpoint that are problematic for production > operations. Will use S3 as an example DFS for the rest of description. > 1. Because there is no way to deterministically know how far back the > incremental checkpoint can refer to files uploaded to S3, it is very > difficult to set S3 bucket/object TTL. In one application, we have observed > Flink checkpoint referring to files uploaded over 6 months ago. S3 TTL can > corrupt the Flink checkpoints. > S3 TTL is important for a few reasons > - purge orphaned files (like external checkpoints from previous deployments) > to keep the storage cost in check. This problem can be addressed by > implementing proper garbage collection (similar to JVM) by traversing the > retained checkpoints from all jobs and traverse the file references. But that > is an expensive solution from engineering cost perspective. > - Security and privacy. E.g., there may be requirement that Flink state can't > keep the data for more than some duration threshold (hours/days/weeks). > Application is expected to purge keys to satisfy the requirement. However, > with incremental checkpoint and how deletion works in RocksDB, it is hard to > set S3 TTL to purge S3 files. Even though those old S3 files don't contain > live keys, they may still be referrenced by retained Flink checkpoints. > 2. Occasionally, corrupted checkpoint files (on S3) are observed. As a > result, restoring from checkpoint failed. With incremental checkpoint, it > usually doesn't help to try other older checkpoints, because they may refer > to the same corrupted file. It is unclear whether the corruption happened > before or during S3 upload. This risk can be mitigated with periodical > savepoints. > It all boils down to periodical full snapshot (checkpoint or savepoint) to > deterministically break the chain of incremental checkpoints. Search the jira > history, the behavior that FLINK-23949 [1] trying to fix is actually close to > what we would need here. > There are a few options > 1. Periodically trigger savepoints (via control plane). This is actually not > a bad practice and might be appealing to some people. The problem is that it > requires a job deployment to break the chain of incremental checkpoint. > periodical job deployment may sound hacky. If we make the behavior of full > checkpoint after a savepoint (fixed in FLINK-23949) configurable, it might be > an acceptable compromise. The benefit is that no job deployment is required > after savepoints. > 2. Build the feature in Flink incremental checkpoint. Periodically (with some > cron style config) trigger a full checkpoint to break the incremental chain. > If the full checkpoint failed (due to whatever reason), the following > checkpoints should attempt full checkpoint as well until one successful full > checkpoint is completed. > 3. For the security/privacy requirement, the main thing is to apply > compaction on the deleted keys. That could probably avoid references to the > old files. Is there any RocksDB compation can achieve full compaction of > removing old delete markers. Recent delete markers are fine > [1] https://issues.apache.org/jira/browse/FLINK-23949 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852: URL: https://github.com/apache/flink/pull/20852#discussion_r974870165 ## flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMasterGateway.java: ## @@ -218,13 +220,27 @@ CompletableFuture triggerSavepoint( final SavepointFormatType formatType, @RpcTimeout final Time timeout); +/** + * Triggers taking a checkpoint of the executed job. + * + * @param checkpointProperties to determine how checkpoint should be taken or null if the + * existing checkpoint property should be used + * @param timeout for the rpc call + * @return Future which is completed with the savepoint path once completed + */ +CompletableFuture triggerCheckpoint( Review Comment: Good question, @pnowojski. Assuming we are talking about 1. `CompletableFuture triggerCheckpoint(@RpcTimeout final Time timeout)` as the other method 2. `CompletedOperationCache checkpointTriggerCache` in the `DispatcherCachedOperationsHandler` as the cache (location as cache value?) We definitely can. Meanwhile I guess this means that the restful API (checkpoint trigger status API) will eventually return checkpoint location in the restful response body, instead of checkpoint id. The reason for returning `CompletedCheckpoint` was to facilitate both old method and new API: the new restful API was returning checkpoint id, while the other method was returning checkpoint location, and CompletedCheckpoint contains both. I am pretty new to flink so I am not sure what are people typically care about a checkpoint. If we think it is a better design to return checkpoint location, instead of checkpoint id in the response body in the restful API, I am happy to return checkpoint location and reuse existing cache and another method. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29345) Too many open files in table store orc writer
Jingsong Lee created FLINK-29345: Summary: Too many open files in table store orc writer Key: FLINK-29345 URL: https://issues.apache.org/jira/browse/FLINK-29345 Project: Flink Issue Type: Bug Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.3.0 Attachments: image-2022-09-20-11-57-11-373.png !image-2022-09-20-11-57-11-373.png! We can avoid reading the local file to obtain the config every time we create a new writer by reusing the prepared configuration. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-25328) Improvement of reuse segments for join/agg/sort operators in TaskManager for flink olap queries
[ https://issues.apache.org/jira/browse/FLINK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606865#comment-17606865 ] Shammon commented on FLINK-25328: - Hi [~xtsong] Sorry for not following this issue for a long time. According to what we discussed above, I have wrote a design doc [Memory Manager Pool Design|https://docs.google.com/document/d/1y8XA7S0BK0-BfRuUp8atIutoHj-2z_5BIxTmL1EmLRM] for this issue and wanted to implement this issue in the next weeks. Could you help to review this doc and feel free to add comments on it, THX > Improvement of reuse segments for join/agg/sort operators in TaskManager for > flink olap queries > --- > > Key: FLINK-25328 > URL: https://issues.apache.org/jira/browse/FLINK-25328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.14.0, 1.12.5, 1.13.3 >Reporter: Shammon >Priority: Major > > We submit batch jobs to flink session cluster as olap queries, and these > jobs' subtasks in TaskManager are frequently created and destroyed because > they finish their work quickly. Each slot in taskmanager manages > `MemoryManager` for multiple tasks in one job, and the `MemoryManager` is > closed when all the subtasks are finished. Join/Aggregate/Sort and etc. > operators in the subtasks allocate `MemorySegment` via `MemoryManager` and > these `MemorySegment` will be free when they are finished. > > It causes too much memory allocation and free of `MemorySegment` in > taskmanager. For example, a TaskManager contains 50 slots, one job has 3 > join/agg operatos run in the slot, each operator will allocate 2000 segments > and initialize them. If the subtasks of a job take 100ms to execute, then the > taskmanager will execute 10 jobs' subtasks one second and it will allocate > and free 2000 * 3 * 50 * 10 = 300w segments for them. Allocate and free too > many segments from memory will cause two issues: > 1) Increases the CPU usage of taskmanager > 2) Increase the cost of subtasks in taskmanager, which will increase the > latency of job and decrease the qps. > To improve the usage of memory segment between jobs in the same slot, > we propose not drop memory manager when all the subtasks in the slot are > finished. The slot will hold the `MemoryManager` and not free the allocated > `MemorySegment` in it immediately. When some subtasks of another job are > assigned to the slot, they don't need to allocate segments from memory and > can reuse the `MemoryManager` and `MemorySegment` in it. WDYT? [~xtsong] THX -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29274) HiveServer2EndpointITCase.testGetFunctionWithPattern failed with Persistence Manager has been closed
[
https://issues.apache.org/jira/browse/FLINK-29274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shengkai Fang updated FLINK-29274:
--
Priority: Blocker (was: Critical)
> HiveServer2EndpointITCase.testGetFunctionWithPattern failed with Persistence
> Manager has been closed
>
>
> Key: FLINK-29274
> URL: https://issues.apache.org/jira/browse/FLINK-29274
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.16.0
>Reporter: Xingbo Huang
>Assignee: yuzelin
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.16.0
>
>
> {code:java}
> 4.6807800Z Sep 13 02:07:54 [ERROR]
> org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.testGetFunctionWithPattern
> Time elapsed: 22.127 s <<< ERROR!
> 2022-09-13T02:07:54.6813586Z Sep 13 02:07:54 java.sql.SQLException:
> javax.jdo.JDOFatalUserException: Persistence Manager has been closed
> 2022-09-13T02:07:54.6815315Z Sep 13 02:07:54 at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:401)
> 2022-09-13T02:07:54.6816917Z Sep 13 02:07:54 at
> org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:266)
> 2022-09-13T02:07:54.6818338Z Sep 13 02:07:54 at
> org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.lambda$testGetFunctionWithPattern$29(HiveServer2EndpointITCase.java:542)
> 2022-09-13T02:07:54.6819988Z Sep 13 02:07:54 at
> org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.runGetObjectTest(HiveServer2EndpointITCase.java:633)
> 2022-09-13T02:07:54.6821484Z Sep 13 02:07:54 at
> org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.runGetObjectTest(HiveServer2EndpointITCase.java:621)
> 2022-09-13T02:07:54.6823318Z Sep 13 02:07:54 at
> org.apache.flink.table.endpoint.hive.HiveServer2EndpointITCase.testGetFunctionWithPattern(HiveServer2EndpointITCase.java:539)
> 2022-09-13T02:07:54.6824711Z Sep 13 02:07:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-09-13T02:07:54.6825817Z Sep 13 02:07:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-09-13T02:07:54.6827003Z Sep 13 02:07:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-09-13T02:07:54.6828259Z Sep 13 02:07:54 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-09-13T02:07:54.6829478Z Sep 13 02:07:54 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:725)
> 2022-09-13T02:07:54.6830717Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
> 2022-09-13T02:07:54.6832444Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
> 2022-09-13T02:07:54.6834028Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:149)
> 2022-09-13T02:07:54.6835304Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:140)
> 2022-09-13T02:07:54.6836734Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:84)
> 2022-09-13T02:07:54.6838257Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
> 2022-09-13T02:07:54.6839775Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
> 2022-09-13T02:07:54.6841400Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> 2022-09-13T02:07:54.6843309Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
> 2022-09-13T02:07:54.6845300Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
> 2022-09-13T02:07:54.6846879Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
> 2022-09-13T02:07:54.6848406Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
> 2022-09-13T02:07:54.6849760Z Sep 13 02:07:54 at
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
> 2022-09-13T02:07:54.6851297Z Sep 13 02:07:54 at
>
[GitHub] [flink] flinkbot commented on pull request #20858: [FLINK-29219][table]fix CREATE TABLE AS statement blocks SQL client's execution
flinkbot commented on PR #20858: URL: https://github.com/apache/flink/pull/20858#issuecomment-1251795225 ## CI report: * ba347c7d86180efd6572d5d3ee6ed732eeb088ee UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29219) CREATE TABLE AS statement blocks SQL client's execution
[ https://issues.apache.org/jira/browse/FLINK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu updated FLINK-29219: -- Fix Version/s: 1.16.0 > CREATE TABLE AS statement blocks SQL client's execution > --- > > Key: FLINK-29219 > URL: https://issues.apache.org/jira/browse/FLINK-29219 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Client >Affects Versions: 1.16.0 >Reporter: Qingsheng Ren >Assignee: dalongliu >Priority: Major > Fix For: 1.16.0 > > > When executing CREATE TABLE AS statement to create a sink table in SQL > client, the client could create the table in catalog and submit the job to > cluster successfully, but stops emitting new prompts and accepts new inputs, > and user has to use SIGTERM (Control + C) to forcefully stop the SQL client. > As contrast the behavior of INSERT INTO statement in SQL client is printing > "Job is submitted with JobID " and being ready to accept user's input. > From the log it looks like the client was waiting for the job to finish. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29219) CREATE TABLE AS statement blocks SQL client's execution
[ https://issues.apache.org/jira/browse/FLINK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29219: --- Labels: pull-request-available (was: ) > CREATE TABLE AS statement blocks SQL client's execution > --- > > Key: FLINK-29219 > URL: https://issues.apache.org/jira/browse/FLINK-29219 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Client >Affects Versions: 1.16.0 >Reporter: Qingsheng Ren >Assignee: dalongliu >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0 > > > When executing CREATE TABLE AS statement to create a sink table in SQL > client, the client could create the table in catalog and submit the job to > cluster successfully, but stops emitting new prompts and accepts new inputs, > and user has to use SIGTERM (Control + C) to forcefully stop the SQL client. > As contrast the behavior of INSERT INTO statement in SQL client is printing > "Job is submitted with JobID " and being ready to accept user's input. > From the log it looks like the client was waiting for the job to finish. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29219) CREATE TABLE AS statement blocks SQL client's execution
[ https://issues.apache.org/jira/browse/FLINK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu updated FLINK-29219: -- Priority: Blocker (was: Major) > CREATE TABLE AS statement blocks SQL client's execution > --- > > Key: FLINK-29219 > URL: https://issues.apache.org/jira/browse/FLINK-29219 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Client >Affects Versions: 1.16.0 >Reporter: Qingsheng Ren >Assignee: dalongliu >Priority: Blocker > Labels: pull-request-available > Fix For: 1.16.0 > > > When executing CREATE TABLE AS statement to create a sink table in SQL > client, the client could create the table in catalog and submit the job to > cluster successfully, but stops emitting new prompts and accepts new inputs, > and user has to use SIGTERM (Control + C) to forcefully stop the SQL client. > As contrast the behavior of INSERT INTO statement in SQL client is printing > "Job is submitted with JobID " and being ready to accept user's input. > From the log it looks like the client was waiting for the job to finish. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Tartarus0zm opened a new pull request, #20858: [FLINK-29219][table]fix CREATE TABLE AS statement blocks SQL client's execution
Tartarus0zm opened a new pull request, #20858: URL: https://github.com/apache/flink/pull/20858 ## What is the purpose of the change * fix CTAS statement blocks SQL client's execution ## Brief change log `ContextResolvedTable` objects are constructed manually and not obtained using `CatalogManager`. ## Verifying this change * The change was verified manually by using sql-client.sh to execute the CTAS statement with `table.dml-sync=false` then the sql client was not blocked. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29344) Make Adaptive Scheduler supports Fine-Grained Resource Management
Xintong Song created FLINK-29344:
Summary: Make Adaptive Scheduler supports Fine-Grained Resource
Management
Key: FLINK-29344
URL: https://issues.apache.org/jira/browse/FLINK-29344
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Reporter: Xintong Song
This ticket is a reflection of the following Slack discussion:
{quote}
Donatien Schmitz
Adaptive Scheduler thread:
Hey all, it seems like the Adaptive Scheduler does not support fine grain
resource management. I have fixed it and would like to know if you would be
interested in a PR or if it was purposely designed to not support Fine grain
resource management.
rmetzger
@Donatien Schmitz: I’m concerned that we don’t have a lot of review capacity
right now, and I’m now aware of any users asking for it.
rmetzger
I couldn’t find a ticket for adding this feature, did you find one?
If not, can you add one? This will allow us to at least making this feature
show up on google, and people might comment on it, if they need it.
rmetzger
If the change is fairly self-contained, is unlikely to cause instabilities,
then we can also consider merging it
rmetzger
@Xintong Song what do you think?
Xintong Song
@rmetzger, thanks for involving me.
@Donatien Schmitz, thanks for bringing this up, and for volunteering on fixing
this. Could you explain a bit more about how do you plan to fix this?
Fine-grained resource management is not yet supported by adaptive scheduler,
because there’s an issue that we haven’t find a good solution for. Namely, if
only part of the resource requirements can be fulfilled, how do we decide which
requirements should be fulfilled. E.g., say the job declares it needs 10 slots
with resource 1 for map tasks, and another 10 slots with resource 2 for reduce
tasks. If there’s not enough resources (say only 10 slots can be allocated for
simplicity), how many slots for map / reduce tasks should be allocated?
Obviously, <10 map, 0 reduce> & <0 map, 10 reduce> would not work. For this
example, a proportional scale-down (<5 map, 5 reduce>) seems reasonable.
However, a proportional scale-down is not always easy (e.g., requirements is
<100 map, 1 reduce>), and the issue grows more complicated if you take lots of
stages and the differences of slot sizes into consideration.
I’d like to see adaptive scheduler also supports fine-grained resource
management. If there’s a good solution to the above issue, I’d love to help
review the effort.
Donatien Schmitz
Dear Robert and Xintong, thanks for reading and reacting to my message! I'll
reply tomorrow (GTM +1 time) if that's quite alright with you. Best, Donatien
Schmitz
Donatien Schmitz
@Xintong Song
* We are working on fine-grain scheduling for resource optimisation of long
running or periodic jobs. One of the feature we are experiencing is a
"rescheduling plan", a mapping of operators and Resource Profiles that can be
dynamically applied to a running job. This rescheduling would be triggered by
policies about some metrics (focus on RocksDB in our case).
* While developing this new feature, we decided to implement it on the Adpative
Scheduler instead of the Base Scheduler because the logic brought by the state
machine already present made it more logical: transitions from states Executing
-> Cancelling -> Rescheduling -> Waiting for Resources -> Creating -> Executing
* In our case we are working on a POC and thus focusing on a real simple job
with a // of 1. The issue you brought is indeed something we have faced while
raising the // of the job.
* If you create a Jira Ticket we can discuss it over there if you'd like!
Donatien Schmitz
@rmetzger The changes do not break the default resource management but does not
fix the issue brought out by Xintong.
{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29344) Make Adaptive Scheduler supports Fine-Grained Resource Management
[
https://issues.apache.org/jira/browse/FLINK-29344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606860#comment-17606860
]
Xintong Song commented on FLINK-29344:
--
cc [~donaschmi] [~rmetzger]
> Make Adaptive Scheduler supports Fine-Grained Resource Management
> -
>
> Key: FLINK-29344
> URL: https://issues.apache.org/jira/browse/FLINK-29344
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Xintong Song
>Priority: Major
>
> This ticket is a reflection of the following Slack discussion:
> {quote}
> Donatien Schmitz
> Adaptive Scheduler thread:
> Hey all, it seems like the Adaptive Scheduler does not support fine grain
> resource management. I have fixed it and would like to know if you would be
> interested in a PR or if it was purposely designed to not support Fine grain
> resource management.
> rmetzger
> @Donatien Schmitz: I’m concerned that we don’t have a lot of review capacity
> right now, and I’m now aware of any users asking for it.
> rmetzger
> I couldn’t find a ticket for adding this feature, did you find one?
> If not, can you add one? This will allow us to at least making this feature
> show up on google, and people might comment on it, if they need it.
> rmetzger
> If the change is fairly self-contained, is unlikely to cause instabilities,
> then we can also consider merging it
> rmetzger
> @Xintong Song what do you think?
> Xintong Song
> @rmetzger, thanks for involving me.
> @Donatien Schmitz, thanks for bringing this up, and for volunteering on
> fixing this. Could you explain a bit more about how do you plan to fix this?
> Fine-grained resource management is not yet supported by adaptive scheduler,
> because there’s an issue that we haven’t find a good solution for. Namely, if
> only part of the resource requirements can be fulfilled, how do we decide
> which requirements should be fulfilled. E.g., say the job declares it needs
> 10 slots with resource 1 for map tasks, and another 10 slots with resource 2
> for reduce tasks. If there’s not enough resources (say only 10 slots can be
> allocated for simplicity), how many slots for map / reduce tasks should be
> allocated? Obviously, <10 map, 0 reduce> & <0 map, 10 reduce> would not work.
> For this example, a proportional scale-down (<5 map, 5 reduce>) seems
> reasonable. However, a proportional scale-down is not always easy (e.g.,
> requirements is <100 map, 1 reduce>), and the issue grows more complicated if
> you take lots of stages and the differences of slot sizes into consideration.
> I’d like to see adaptive scheduler also supports fine-grained resource
> management. If there’s a good solution to the above issue, I’d love to help
> review the effort.
> Donatien Schmitz
> Dear Robert and Xintong, thanks for reading and reacting to my message! I'll
> reply tomorrow (GTM +1 time) if that's quite alright with you. Best, Donatien
> Schmitz
> Donatien Schmitz
> @Xintong Song
> * We are working on fine-grain scheduling for resource optimisation of long
> running or periodic jobs. One of the feature we are experiencing is a
> "rescheduling plan", a mapping of operators and Resource Profiles that can be
> dynamically applied to a running job. This rescheduling would be triggered by
> policies about some metrics (focus on RocksDB in our case).
> * While developing this new feature, we decided to implement it on the
> Adpative Scheduler instead of the Base Scheduler because the logic brought by
> the state machine already present made it more logical: transitions from
> states Executing -> Cancelling -> Rescheduling -> Waiting for Resources ->
> Creating -> Executing
> * In our case we are working on a POC and thus focusing on a real simple job
> with a // of 1. The issue you brought is indeed something we have faced while
> raising the // of the job.
> * If you create a Jira Ticket we can discuss it over there if you'd like!
> Donatien Schmitz
> @rmetzger The changes do not break the default resource management but does
> not fix the issue brought out by Xintong.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29343) Fix fail to execute ddl in HiveDialect when use specifc catalog in sql statement
luoyuxia created FLINK-29343: Summary: Fix fail to execute ddl in HiveDialect when use specifc catalog in sql statement Key: FLINK-29343 URL: https://issues.apache.org/jira/browse/FLINK-29343 Project: Flink Issue Type: Bug Components: Connectors / Hive Affects Versions: 1.16.0 Reporter: luoyuxia Fix For: 1.16.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29339) JobMasterPartitionTrackerImpl#requestShuffleDescriptorsFromResourceManager blocks main thread
[
https://issues.apache.org/jira/browse/FLINK-29339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606839#comment-17606839
]
Yun Gao commented on FLINK-29339:
-
[~xuannan] Could you have a look at this issue~?
> JobMasterPartitionTrackerImpl#requestShuffleDescriptorsFromResourceManager
> blocks main thread
> -
>
> Key: FLINK-29339
> URL: https://issues.apache.org/jira/browse/FLINK-29339
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0
>Reporter: Chesnay Schepler
>Priority: Critical
>
> {code:java}
> private List requestShuffleDescriptorsFromResourceManager(
> IntermediateDataSetID intermediateDataSetID) {
> Preconditions.checkNotNull(
> resourceManagerGateway, "JobMaster is not connected to
> ResourceManager");
> try {
> return this.resourceManagerGateway
> .getClusterPartitionsShuffleDescriptors(intermediateDataSetID)
> .get(); // <-- there's your problem
> } catch (Throwable e) {
> throw new RuntimeException(
> String.format(
> "Failed to get shuffle descriptors of intermediate
> dataset %s from ResourceManager",
> intermediateDataSetID),
> e);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20857: [hotfix] Fix the problem that BatchShuffleItCase not subject to configuration.
flinkbot commented on PR #20857: URL: https://github.com/apache/flink/pull/20857#issuecomment-1251774864 ## CI report: * 6e4307aec8052d54a6e84b291f51a6ce70025b2b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kylemeow commented on a diff in pull request #20542: [FLINK-28910][Connectors/hbase]Fix potential data deletion while updating HBase rows
kylemeow commented on code in PR #20542:
URL: https://github.com/apache/flink/pull/20542#discussion_r974839709
##
flink-connectors/flink-connector-hbase-base/src/main/java/org/apache/flink/connector/hbase/sink/HBaseSinkFunction.java:
##
@@ -191,8 +195,10 @@ private void checkErrorAndRethrow() {
public void invoke(T value, Context context) throws Exception {
checkErrorAndRethrow();
-mutator.mutate(mutationConverter.convertToMutation(value));
-
+Mutation mutation = mutationConverter.convertToMutation(value);
+synchronized (mutationMap) {
+mutationMap.put(mutation.getRow(), mutation);
Review Comment:
One potential bug might arise when the `mutation.getRow()` returns an array.
As we know, the `hashCode` and `equals` of two different array instances are
different regardless of whether their contents are identical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] reswqa opened a new pull request, #20857: [hotfix] Fix the problem that BatchShuffleItCase not subject to configuration.
reswqa opened a new pull request, #20857: URL: https://github.com/apache/flink/pull/20857 ## What is the purpose of the change *Fix the problem that `BatchShuffleItCase` not subject to configuration.* ## Brief change log - *pass configuration to `StreamExecutionEnvironment.getExecutionEnvironment`* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on a diff in pull request #20776: [FLINK-29185][hive] fix class not found exception for creating temporary function using jar with Hive dialect
wuchong commented on code in PR #20776:
URL: https://github.com/apache/flink/pull/20776#discussion_r974834238
##
flink-connectors/flink-connector-hive/src/test/java/org/apache/flink/connectors/hive/HiveDialectITCase.java:
##
@@ -745,32 +745,40 @@ public void testTemporaryFunctionUDAF() throws Exception {
public void testCreateFunctionUsingJar() throws Exception {
tableEnv.executeSql("create table src(x int)");
tableEnv.executeSql("insert into src values (1), (2)").await();
-String udfClass = "addOne";
-String udfCode =
-"public class "
-+ udfClass
+String udfCodeTemplate =
+"public class %s"
+ " extends org.apache.hadoop.hive.ql.exec.UDF {\n"
+ " public int evaluate(int content) {\n"
+ "return content + 1;\n"
+ " }"
+ "}\n";
+String udfClass = "addOne";
+String udfCode = String.format(udfCodeTemplate, udfClass);
File jarFile =
UserClassLoaderJarTestUtils.createJarFile(
tempFolder.newFolder("test-jar"), "test-udf.jar",
udfClass, udfCode);
// test create function using jar
tableEnv.executeSql(
String.format(
-"create function add_one as 'addOne' using jar '%s'",
jarFile.getPath()));
+"create function add_one as '%s' using jar '%s'",
+udfClass, jarFile.getPath()));
assertThat(
CollectionUtil.iteratorToList(
tableEnv.executeSql("select add_one(x)
from src").collect())
.toString())
.isEqualTo("[+I[2], +I[3]]");
+
// test create temporary function using jar
+// create a new jarfile with a new class name
+udfClass = "addOne1";
+udfCode = String.format(udfCodeTemplate, udfClass);
Review Comment:
It seems there is a flaw in the current design/implementation that the
previous user jar is not removed from the classloader which may lead to class
conflict when using the same class name but a different code. For example, if
the code of `addOne1` is `content - 1`, the executed result might still be
`content + 1`. @lsyldliu could you create an issue to track that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #295: [FLINK-29297] Group Table Store file writers into SingleFileWriter and RollingFileWriter
JingsongLi commented on code in PR #295:
URL: https://github.com/apache/flink-table-store/pull/295#discussion_r974827083
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/io/KeyValueDataRollingFileWriter.java:
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.file.io;
+
+import org.apache.flink.table.store.file.KeyValue;
+import org.apache.flink.table.store.file.data.DataFileMeta;
+
+import java.util.function.Supplier;
+
+/** A {@link RollingFileWriter} to write {@link KeyValue}s into several
rolling data files. */
+public class KeyValueDataRollingFileWriter extends RollingFileWriter {
Review Comment:
Do we need to create separate class? I think there is no logical.
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/io/SingleFileWriter.java:
##
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.table.store.file.io;
+
+import org.apache.flink.api.common.serialization.BulkWriter;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.store.file.utils.FileUtils;
+import org.apache.flink.util.IOUtils;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.function.Function;
+
+/**
+ * A {@link FileWriter} to produce a single file.
+ *
+ * @param type of records to write.
+ * @param type of result to produce after writing a file.
+ */
+public abstract class SingleFileWriter implements FileWriter {
+
+private static final Logger LOG =
LoggerFactory.getLogger(SingleFileWriter.class);
+
+private final BulkWriter.Factory factory;
+protected final Path path;
+private final Function converter;
+
+private long recordCount;
+private long length;
+protected boolean closed;
+
+private BulkWriter writer;
+private FSDataOutputStream out;
+
+public SingleFileWriter(
+BulkWriter.Factory factory, Path path, Function converter) {
+this.factory = factory;
+this.path = path;
+this.converter = converter;
+
+this.recordCount = 0;
+this.length = 0;
+this.closed = false;
+
+this.writer = null;
+this.out = null;
+}
+
+public Path path() {
+return path;
+}
+
+@Override
+public void write(T record) throws IOException {
+writeImpl(record);
+}
+
+protected RowData writeImpl(T record) throws IOException {
+if (closed) {
+throw new RuntimeException("Writer has already closed!");
+}
+
+try {
+if (writer == null) {
+writer = createWriter();
+}
+RowData rowData = converter.apply(record);
+writer.addElement(rowData);
+recordCount++;
+return rowData;
+} catch (Throwable e) {
+LOG.warn("Exception occurs when writing file " + path + ".
Cleaning up.", e);
+abort();
+throw e;
+}
+}
+
+@Override
+public long recordCount() {
+return recordCount;
+}
+
+@Override
+public long length() throws IOException {
+if (closed) {
+return length;
+}
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #373: [FLINK-28574] Bump the JOSDK version to 3.2.2
morhidi commented on PR #373: URL: https://github.com/apache/flink-kubernetes-operator/pull/373#issuecomment-1251760723 +1 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29342) Do not query writer.length() per record in RollingFileWriter
[ https://issues.apache.org/jira/browse/FLINK-29342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chan updated FLINK-29342: -- Description: It is expensive for the local file system to get pos. (was: It is expensive for local file system to do so.) > Do not query writer.length() per record in RollingFileWriter > > > Key: FLINK-29342 > URL: https://issues.apache.org/jira/browse/FLINK-29342 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jane Chan >Priority: Major > Fix For: table-store-0.3.0 > > > It is expensive for the local file system to get pos. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29342) Do not query writer.length() per record in RollingFileWriter
[ https://issues.apache.org/jira/browse/FLINK-29342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chan updated FLINK-29342: -- Description: It is expensive for local file system to do so. > Do not query writer.length() per record in RollingFileWriter > > > Key: FLINK-29342 > URL: https://issues.apache.org/jira/browse/FLINK-29342 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: table-store-0.3.0 >Reporter: Jane Chan >Priority: Major > Fix For: table-store-0.3.0 > > > It is expensive for local file system to do so. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29342) Do not query writer.length() per record in RollingFileWriter
Jane Chan created FLINK-29342: - Summary: Do not query writer.length() per record in RollingFileWriter Key: FLINK-29342 URL: https://issues.apache.org/jira/browse/FLINK-29342 Project: Flink Issue Type: Improvement Components: Table Store Affects Versions: table-store-0.3.0 Reporter: Jane Chan Fix For: table-store-0.3.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] SteNicholas commented on pull request #372: [FLINK-29261] Use FAIL_ON_UNKNOWN_PROPERTIES=false in ReconciliationUtils
SteNicholas commented on PR #372: URL: https://github.com/apache/flink-kubernetes-operator/pull/372#issuecomment-1251742542 @morhidi @gyfora when using `FAIL_ON_UNKNOWN_PROPERTIES=false`, is it necessary to tell the users to make sure that the correct json properties have already verified and guaranteed by Kubernetes itself based on the installed CRD on the document? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29025) Add doc for Hive Dialect
[ https://issues.apache.org/jira/browse/FLINK-29025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-29025. --- Fix Version/s: 1.17.0 Assignee: luoyuxia Resolution: Fixed Fixed in - master: 99e892736178993ad2beafcec7a44c0f4d507758..7c9f5ec1845c4a2332ebda944d3ef6e3a805d1a3 - release-1.16: 53194166ee70c71aae55bedf1030f44aaf50ca3d..100979d89613ca30b07c905fc8b3c444aa43e3c0 > Add doc for Hive Dialect > > > Key: FLINK-29025 > URL: https://issues.apache.org/jira/browse/FLINK-29025 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive, Documentation >Affects Versions: 1.16.0 >Reporter: luoyuxia >Assignee: luoyuxia >Priority: Major > Labels: pull-request-available > Fix For: 1.16.0, 1.17.0 > > > Moving some stuff from connectors/table/hive/hive_dialect. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wuchong merged pull request #20696: [FLINK-29025][docs] add docs for Hive dialect
wuchong merged PR #20696: URL: https://github.com/apache/flink/pull/20696 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong merged pull request #20856: [BP-1.16][FLINK-29025][docs] add docs for Hive dialect
wuchong merged PR #20856: URL: https://github.com/apache/flink/pull/20856 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] SteNicholas opened a new pull request, #373: [FLINK-28574] Bump the JOSDK version to 3.2.2
SteNicholas opened a new pull request, #373: URL: https://github.com/apache/flink-kubernetes-operator/pull/373 ## What is the purpose of the change Java Operator SDK has released [3.2.2](https://github.com/java-operator-sdk/java-operator-sdk/releases/tag/v3.2.2) version, which fixes config service override executor service concurrent reconciliation numbers . Bump the JOSDK version to 3.2.2. cc @csviri ## Brief change log - Bump the JOSDK version to 3.2.2 ## Verifying this change ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changes to the `CustomResourceDescriptors`: (yes / **no**) - Core observer or reconciler logic that is regularly executed: (yes / **no**) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #155: [FLINK-29318] Add Transformer for PolynomialExpansion
yunfengzhou-hub commented on code in PR #155: URL: https://github.com/apache/flink-ml/pull/155#discussion_r974805771 ## docs/content/docs/operators/feature/polynomialexpansion.md: ## @@ -0,0 +1,160 @@ +--- +title: "PolynomialExpansion" +weight: 1 +type: docs +aliases: +- /operators/feature/polynomialexpansion.html +--- + + + +## PolynomialExpansion + +A Transformer that expands the input vectors in polynomial space. + +Take a 2-dimension vector as an example: `(x, y)`, if we want to expand it with degree 2, then +we get `(x, x * x, y, x * y, y * y)`. + +For more information about the polynomial expansion, see +http://en.wikipedia.org/wiki/Polynomial_expansion. + +### Input Columns + +| Param name | Type | Default | Description | +|:---|:---|:--|:--| +| inputCol | Vector | `"input"` | Vectors to be normalized. | Review Comment: nit: not to be normalized, but to be expanded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-26999) Introduce ClickHouse Connector
[ https://issues.apache.org/jira/browse/FLINK-26999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606816#comment-17606816 ] Kanthi Subramanian commented on FLINK-26999: Is this being worked on, would love to contribute. > Introduce ClickHouse Connector > -- > > Key: FLINK-26999 > URL: https://issues.apache.org/jira/browse/FLINK-26999 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common >Affects Versions: 1.15.0 >Reporter: ZhuoYu Chen >Priority: Major > > [https://cwiki.apache.org/confluence/display/FLINK/FLIP-202%3A+Introduce+ClickHouse+Connector] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-27510) update playgrounds for Flink 1.15
[ https://issues.apache.org/jira/browse/FLINK-27510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Anderson resolved FLINK-27510. Resolution: Implemented > update playgrounds for Flink 1.15 > - > > Key: FLINK-27510 > URL: https://issues.apache.org/jira/browse/FLINK-27510 > Project: Flink > Issue Type: Improvement > Components: Documentation / Training / Exercises >Affects Versions: 1.15.0 >Reporter: David Anderson >Assignee: Roy Michael >Priority: Minor > Labels: auto-deprioritized-major, starter > Fix For: 1.15.2 > > > All of the playgrounds should be updated for Flink 1.15. This should include > reworking the code as necessary to avoid using anything that has been > deprecated. > It would be best to first perform the updates for 1.14, so that APIs newly > deprecated in 1.14 can be fixed first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-27513) Update table walkthrough playground for 1.15
[ https://issues.apache.org/jira/browse/FLINK-27513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Anderson resolved FLINK-27513. Resolution: Implemented master: 677a4bb892f6eba990617a8ecc699b14b10e7b6a > Update table walkthrough playground for 1.15 > > > Key: FLINK-27513 > URL: https://issues.apache.org/jira/browse/FLINK-27513 > Project: Flink > Issue Type: Sub-task > Components: Documentation / Training / Exercises >Reporter: David Anderson >Assignee: Roy Michael >Priority: Major > Labels: pull-request-available, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27510) update playgrounds for Flink 1.15
[ https://issues.apache.org/jira/browse/FLINK-27510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Anderson updated FLINK-27510: --- Fix Version/s: 1.15.2 > update playgrounds for Flink 1.15 > - > > Key: FLINK-27510 > URL: https://issues.apache.org/jira/browse/FLINK-27510 > Project: Flink > Issue Type: Improvement > Components: Documentation / Training / Exercises >Affects Versions: 1.15.0 >Reporter: David Anderson >Assignee: Roy Michael >Priority: Minor > Labels: auto-deprioritized-major, starter > Fix For: 1.15.2 > > > All of the playgrounds should be updated for Flink 1.15. This should include > reworking the code as necessary to avoid using anything that has been > deprecated. > It would be best to first perform the updates for 1.14, so that APIs newly > deprecated in 1.14 can be fixed first. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-27512) Update pyflink walkthrough playground for 1.15
[ https://issues.apache.org/jira/browse/FLINK-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Anderson resolved FLINK-27512. Resolution: Implemented master: 7d55cc567bb2c064f7ce49ff21faa9486bd3de65 > Update pyflink walkthrough playground for 1.15 > -- > > Key: FLINK-27512 > URL: https://issues.apache.org/jira/browse/FLINK-27512 > Project: Flink > Issue Type: Sub-task > Components: Documentation / Training / Exercises >Reporter: David Anderson >Assignee: Roy Michael >Priority: Major > Labels: pull-request-available, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-27511) Update operations playground for 1.15
[ https://issues.apache.org/jira/browse/FLINK-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Anderson resolved FLINK-27511. Resolution: Resolved master: c173ab01f20ba1d168657e36cebd64c16238b1d1 > Update operations playground for 1.15 > - > > Key: FLINK-27511 > URL: https://issues.apache.org/jira/browse/FLINK-27511 > Project: Flink > Issue Type: Sub-task > Components: Documentation / Training / Exercises >Reporter: David Anderson >Assignee: Roy Michael >Priority: Major > Labels: pull-request-available, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-playgrounds] alpinegizmo closed pull request #35: [FLINK-27513][flink-playgrounds] Update table walkthrough playground for 1.15
alpinegizmo closed pull request #35: [FLINK-27513][flink-playgrounds] Update table walkthrough playground for 1.15 URL: https://github.com/apache/flink-playgrounds/pull/35 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-playgrounds] alpinegizmo commented on pull request #35: [FLINK-27513][flink-playgrounds] Update table walkthrough playground for 1.15
alpinegizmo commented on PR #35: URL: https://github.com/apache/flink-playgrounds/pull/35#issuecomment-1251683148 Merged with 677a4bb892f6eba990617a8ecc699b14b10e7b6a. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] tedhtchang commented on pull request #313: [FLINK-27852][docs] OLM installation and development documentation
tedhtchang commented on PR #313: URL: https://github.com/apache/flink-kubernetes-operator/pull/313#issuecomment-1251682608 @gyfora Let's do that. Should I create a discussion on the dev list ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29341) Upgrade Apache Kafka version to 7.1.1-ccs to resolve CVE-2021-38153
Sanjay Nathani created FLINK-29341: -- Summary: Upgrade Apache Kafka version to 7.1.1-ccs to resolve CVE-2021-38153 Key: FLINK-29341 URL: https://issues.apache.org/jira/browse/FLINK-29341 Project: Flink Issue Type: Bug Reporter: Sanjay Nathani Fix For: 1.13.6 The flink-connector-kafka module has Kafka as dependency being intorduced from here [https://github.com/apache/flink/blob/release-1.13.6/flink-connectors/flink-connector-kafka/pom.xml] . The version of kafka is 2.4.1 which is vulnerable having CVE-2021-38153 . In order to remove this CVE kafka version should be upgraded to 2.6.3 as said here https://lists.apache.org/thread/7vrvjt7tm7m46txds3kt6bywd8vp5px0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29341) Upgrade Apache Kafka version to 2.6.3 to resolve CVE-2021-38153
[ https://issues.apache.org/jira/browse/FLINK-29341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Nathani updated FLINK-29341: --- Summary: Upgrade Apache Kafka version to 2.6.3 to resolve CVE-2021-38153 (was: Upgrade Apache Kafka version to 7.1.1-ccs to resolve CVE-2021-38153) > Upgrade Apache Kafka version to 2.6.3 to resolve CVE-2021-38153 > --- > > Key: FLINK-29341 > URL: https://issues.apache.org/jira/browse/FLINK-29341 > Project: Flink > Issue Type: Bug >Reporter: Sanjay Nathani >Priority: Major > Fix For: 1.13.6 > > > The flink-connector-kafka module has Kafka as dependency being intorduced > from here > [https://github.com/apache/flink/blob/release-1.13.6/flink-connectors/flink-connector-kafka/pom.xml] > . The version of kafka is 2.4.1 which is vulnerable having CVE-2021-38153 . > In order to remove this CVE kafka version should be upgraded to 2.6.3 as said > here https://lists.apache.org/thread/7vrvjt7tm7m46txds3kt6bywd8vp5px0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-playgrounds] alpinegizmo merged pull request #36: [FLINK-27512][flink-playgrounds] Update pyflink walkthrough playground for 1.15
alpinegizmo merged PR #36: URL: https://github.com/apache/flink-playgrounds/pull/36 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-playgrounds] roy-michael commented on a diff in pull request #35: [FLINK-27513][flink-playgrounds] Update table walkthrough playground for 1.15
roy-michael commented on code in PR #35:
URL: https://github.com/apache/flink-playgrounds/pull/35#discussion_r974607943
##
table-walkthrough/pom.xml:
##
@@ -72,9 +72,15 @@ under the License.
${flink.version}
test
+
+ org.apache.flink
+ flink-table-planner_${scala.binary.version}
+ ${flink.version}
+ test
+
Review Comment:
this works as well. thanks for the clarification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #372: [FLINK-29261] Use FAIL_ON_UNKNOWN_PROPERTIES=false in ReconciliationUtils
gyfora commented on PR #372: URL: https://github.com/apache/flink-kubernetes-operator/pull/372#issuecomment-1251464380 > > There are other components that use `ObjectMapper` the `FlinkValidator` for instance should probably also need this config otherwise the webhook might not be compatible > > I've been thinking about this and I guess this is the only place we want to deserialize an object with unknown fields, otherwise there's no other way for rolling back to an older operator version. I don't see a reason yet we should allow this in webhook. I'm convincible though :) 1. You deploy a new operator version that has a new flinkSpec field. 2. You set that field in a CR 3. You roll back the operator 4. Any change to the CR will call the webhook with the CR containing the now unknown field you set in 2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #371: [FLINK-29313] Observe spec should include latest operator configs + improved test coverage
morhidi commented on PR #371: URL: https://github.com/apache/flink-kubernetes-operator/pull/371#issuecomment-1251456662 Thanks, will test this tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #372: [FLINK-29261] Use FAIL_ON_UNKNOWN_PROPERTIES=false in ReconciliationUtils
morhidi commented on PR #372: URL: https://github.com/apache/flink-kubernetes-operator/pull/372#issuecomment-1251450623 > There are other components that use `ObjectMapper` the `FlinkValidator` for instance should probably also need this config otherwise the webhook might not be compatible I've been thinking about this and I guess this is the only place we want to deserialize an object with unknown fields, otherwise there's no other way for rolling back to an older operator version. I don't see a reason yet we should allow this in webhook. I'm convincible though :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29340) ResourceManagerTest#testProcessResourceRequirementsWhenRecoveryFinished prone to race condition
Chesnay Schepler created FLINK-29340:
Summary:
ResourceManagerTest#testProcessResourceRequirementsWhenRecoveryFinished prone
to race condition
Key: FLINK-29340
URL: https://issues.apache.org/jira/browse/FLINK-29340
Project: Flink
Issue Type: Technical Debt
Components: Runtime / Coordination, Tests
Affects Versions: 1.16.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
The test incorrectly assumes that the {{declareRequiredResources}} has already
been run when calling {{runInMainThread}}, while the RPC could still be in
flight.
This can result in the test failing because within runInMainThread the test
assumes that completing the readyToServeFuture will immediately result in the
processing of resources, due to this workflow having been set up within
delcareRequiredResources. Without it it will just fail because the completion
of the future has in practice no effect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] dannycranmer commented on pull request #20853: [FLINK-29324] [Connectors/Kinesis] Fix NPE for Kinesis connector when closing
dannycranmer commented on PR #20853: URL: https://github.com/apache/flink/pull/20853#issuecomment-1251442562 @flinkbot run azur -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-29324) Calling Kinesis connector close method before subtask starts running results in NPE
[
https://issues.apache.org/jira/browse/FLINK-29324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer reassigned FLINK-29324:
-
Assignee: Dongming.Hu
> Calling Kinesis connector close method before subtask starts running results
> in NPE
> ---
>
> Key: FLINK-29324
> URL: https://issues.apache.org/jira/browse/FLINK-29324
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.14.5, 1.15.2
>Reporter: Anthony Pounds-Cornish
>Assignee: Dongming.Hu
>Priority: Major
> Labels: pull-request-available
>
> When a Flink application is stopped before a Kinesis connector subtask has
> been started, the following exception is thrown:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.close(FlinkKinesisConsumer.java:421)
> ...{noformat}
> This appears to be related to the fact that [fetcher
> creation|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L307]
> may not occur by [the time it is referenced when the consumer is
> closed|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L421].
> A suggested fix is to make the {{close()}} method null safe [as it has been
> in the {{cancel()}}
> method|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L407].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29324) Calling Kinesis connector close method before subtask starts running results in NPE
[
https://issues.apache.org/jira/browse/FLINK-29324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606709#comment-17606709
]
Danny Cranmer commented on FLINK-29324:
---
Thanks [~xiaohei]
> Calling Kinesis connector close method before subtask starts running results
> in NPE
> ---
>
> Key: FLINK-29324
> URL: https://issues.apache.org/jira/browse/FLINK-29324
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.14.5, 1.15.2
>Reporter: Anthony Pounds-Cornish
>Priority: Major
> Labels: pull-request-available
>
> When a Flink application is stopped before a Kinesis connector subtask has
> been started, the following exception is thrown:
> {noformat}
> java.lang.NullPointerException
> at
> org.apache.flink.streaming.connectors.kinesis.FlinkKinesisConsumer.close(FlinkKinesisConsumer.java:421)
> ...{noformat}
> This appears to be related to the fact that [fetcher
> creation|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L307]
> may not occur by [the time it is referenced when the consumer is
> closed|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L421].
> A suggested fix is to make the {{close()}} method null safe [as it has been
> in the {{cancel()}}
> method|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/FlinkKinesisConsumer.java#L407].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-playgrounds] alpinegizmo closed pull request #33: Adding a missing test dependency
alpinegizmo closed pull request #33: Adding a missing test dependency URL: https://github.com/apache/flink-playgrounds/pull/33 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-playgrounds] alpinegizmo commented on a diff in pull request #35: [FLINK-27513][flink-playgrounds] Update table walkthrough playground for 1.15
alpinegizmo commented on code in PR #35:
URL: https://github.com/apache/flink-playgrounds/pull/35#discussion_r974476284
##
README.md:
##
@@ -13,7 +13,7 @@ Currently, the following playgrounds are available:
Flink job. The playground is presented in detail in
["Flink Operations
Playground"](https://ci.apache.org/projects/flink/flink-docs-release-1.14/docs/try-flink/flink-operations-playground),
which is part of the _Try Flink_ section of the Flink documentation.
-* The **Table Walkthrough** (in the `table-walkthrough` folder) shows to use
the Table API to build an analytics pipeline that reads streaming data from
Kafka and writes results to MySQL, along with a real-time dashboard in Grafana.
The walkthrough is presented in detail in ["Real Time Reporting with the Table
API"](https://ci.apache.org/projects/flink/flink-docs-release-1.14/docs/try-flink/table_api),
which is part of the _Try Flink_ section of the Flink documentation.
+* The **Table Walkthrough** (in the `table-walkthrough` folder) shows to use
the Table API to build an analytics pipeline that reads streaming data from
Kafka and writes results to MySQL, along with a real-time dashboard in Grafana.
The walkthrough is presented in detail in ["Real Time Reporting with the Table
API"](https://ci.apache.org/projects/flink/flink-docs-release-1.15/docs/try-flink/table_api),
which is part of the _Try Flink_ section of the Flink documentation.
Review Comment:
```suggestion
* The **Table Walkthrough** (in the `table-walkthrough` folder) shows how to
use the Table API to build an analytics pipeline that reads streaming data from
Kafka and writes results to MySQL, along with a real-time dashboard in Grafana.
The walkthrough is presented in detail in ["Real Time Reporting with the Table
API"](https://ci.apache.org/projects/flink/flink-docs-release-1.15/docs/try-flink/table_api),
which is part of the _Try Flink_ section of the Flink documentation.
```
##
table-walkthrough/pom.xml:
##
@@ -72,9 +72,15 @@ under the License.
${flink.version}
test
+
+ org.apache.flink
+ flink-table-planner_${scala.binary.version}
+ ${flink.version}
+ test
+
Review Comment:
Try this instead -- I think it should work. We want to encourage using the
scala-free versions wherever possible.
```suggestion
org.apache.flink
flink-table-planner-loader
${flink.version}
provided
org.apache.flink
flink-table-runtime
${flink.version}
provided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a diff in pull request #20757: [FLINK-27919] Add FLIP-27-based source for data generation (FLIP-238)
stevenzwu commented on code in PR #20757:
URL: https://github.com/apache/flink/pull/20757#discussion_r974416546
##
flink-tests/src/test/java/org/apache/flink/api/connector/source/lib/util/RateLimitedSourceReaderITCase.java:
##
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.connector.source.lib.util;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceReaderFactory;
+import org.apache.flink.api.connector.source.datagen.DataGeneratorSource;
+import org.apache.flink.api.connector.source.datagen.GeneratorFunction;
+import org.apache.flink.api.connector.source.lib.NumberSequenceSource;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** An integration test for rate limiting built into the DataGeneratorSource.
*/
+public class RateLimitedSourceReaderITCase extends TestLogger {
Review Comment:
Can we add a unit test for the `GatedRateLimiter` with per checkpoint
quota/throttle with source parallelism at 2+? I guess the challenge is on how
to validate the behavior. One option is to have a map operator to calculate
number of elements it received per checkpoint cycle and emit an output as tuple
.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #360: [FLINK-29165][docs] update docs to create and delete deployment via code
morhidi commented on PR #360: URL: https://github.com/apache/flink-kubernetes-operator/pull/360#issuecomment-1251309340 cc @gyfora could you please kick the workflow on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] morhidi commented on pull request #360: [FLINK-29165][docs] update docs to create and delete deployment via code
morhidi commented on PR #360: URL: https://github.com/apache/flink-kubernetes-operator/pull/360#issuecomment-1251307052 +1 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gauravmiglanid11 commented on pull request #360: [FLINK-29165][docs] update docs to create and delete deployment via code
gauravmiglanid11 commented on PR #360: URL: https://github.com/apache/flink-kubernetes-operator/pull/360#issuecomment-1251287060 Hi @morhidi , changes done, please review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #371: [FLINK-29313] Observe spec should include latest operator configs + improved test coverage
gyfora commented on PR #371: URL: https://github.com/apache/flink-kubernetes-operator/pull/371#issuecomment-1251252453 In the long run we probably need a small overhaul of the config handling to eliminate all weird corner cases and special handling. I opened a jira ticket and will work on this for 1.3.0: https://issues.apache.org/jira/browse/FLINK-29338 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29339) JobMasterPartitionTrackerImpl#requestShuffleDescriptorsFromResourceManager blocks main thread
Chesnay Schepler created FLINK-29339:
Summary:
JobMasterPartitionTrackerImpl#requestShuffleDescriptorsFromResourceManager
blocks main thread
Key: FLINK-29339
URL: https://issues.apache.org/jira/browse/FLINK-29339
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.16.0
Reporter: Chesnay Schepler
{code:java}
private List requestShuffleDescriptorsFromResourceManager(
IntermediateDataSetID intermediateDataSetID) {
Preconditions.checkNotNull(
resourceManagerGateway, "JobMaster is not connected to
ResourceManager");
try {
return this.resourceManagerGateway
.getClusterPartitionsShuffleDescriptors(intermediateDataSetID)
.get(); // <-- there's your problem
} catch (Throwable e) {
throw new RuntimeException(
String.format(
"Failed to get shuffle descriptors of intermediate
dataset %s from ResourceManager",
intermediateDataSetID),
e);
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #372: [FLINK-29261] Use FAIL_ON_UNKNOWN_PROPERTIES=false in ReconciliationUtils
gyfora commented on PR #372: URL: https://github.com/apache/flink-kubernetes-operator/pull/372#issuecomment-1251248735 There are other components that use `ObjectMapper` the `FlinkValidator` for instance should probably also need this config otherwise the webhook might not be compatible -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29338) Try to unify observe/deploy config handling in the operator
Gyula Fora created FLINK-29338: -- Summary: Try to unify observe/deploy config handling in the operator Key: FLINK-29338 URL: https://issues.apache.org/jira/browse/FLINK-29338 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Gyula Fora Assignee: Gyula Fora The operator currently juggles different configuration versions in the observer / reconcile loops : observe/deploy config. The way they are used in certain cases can lead to problems or unexpected behaviour and is complex to understand. We can probably eliminate the use of observe config or at least try to unify it as much as possible for a more straightforward behaviour. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29261) Use FAIL_ON_UNKNOWN_PROPERTIES=false in ReconciliationUtils
[ https://issues.apache.org/jira/browse/FLINK-29261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29261: --- Labels: pull-request-available (was: ) > Use FAIL_ON_UNKNOWN_PROPERTIES=false in ReconciliationUtils > --- > > Key: FLINK-29261 > URL: https://issues.apache.org/jira/browse/FLINK-29261 > Project: Flink > Issue Type: Bug >Reporter: Matyas Orhidi >Priority: Major > Labels: pull-request-available > > The operator cannot be downgraded, once the CR specification is written to > the `status` > > Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: > Unrecognized field "mode" (class > org.apache.flink.kubernetes.operator.crd.spec.FlinkDeploymentSpec), not > marked as ignorable (12 known properties: "restartNonce", "imagePullPolicy", > "ingress", "flinkConfiguration", "serviceAccount", "image", "job", > "podTemplate", "jobManager", "logConfiguration", "flinkVersion", > "taskManager"]) > at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: > org.apache.flink.kubernetes.operator.crd.spec.FlinkDeploymentSpec["mode"]) > at > com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61) > at > com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1127) > at > com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1989) > at > com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1700) > at > com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1678) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:319) > at > com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:176) > at > com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322) > at > com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4650) > at > com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2831) > at > com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:3295) > at > org.apache.flink.kubernetes.operator.reconciler.ReconciliationUtils.deserializeSpecWithMeta(ReconciliationUtils.java:288) > ... 18 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] morhidi opened a new pull request, #372: [FLINK-29261] Use FAIL_ON_UNKNOWN_PROPERTIES=false in ReconciliationUtils
morhidi opened a new pull request, #372: URL: https://github.com/apache/flink-kubernetes-operator/pull/372 ## What is the purpose of the change The operator cannot be downgraded with new CRD, once the CR containing a new field is written to the last reconciled/stable spec. ## Brief change log The `ObjectMapper` is modified in `ReconciliationUtils` ``` private static final ObjectMapper objectMapper = new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); ``` ## Verifying this change Added new `ReconciliationUtilsTest.testSpecDeserializationWithUnknownField()` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: yes ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #313: [FLINK-27852][docs] OLM installation and development documentation
gyfora commented on PR #313: URL: https://github.com/apache/flink-kubernetes-operator/pull/313#issuecomment-1251232047 Should we close this PR until there was a discussion and consensus on the dev list? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29257) Kubernetes operator docs are vague about upgrade mode
[ https://issues.apache.org/jira/browse/FLINK-29257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-29257. -- Fix Version/s: kubernetes-operator-1.2.0 Resolution: Fixed merged to main f8d85b0c654334b1892f246427f0b1ac2cdfa018 > Kubernetes operator docs are vague about upgrade mode > - > > Key: FLINK-29257 > URL: https://issues.apache.org/jira/browse/FLINK-29257 > Project: Flink > Issue Type: Improvement > Components: Documentation, Kubernetes Operator >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Minor > Labels: pull-request-available > Fix For: kubernetes-operator-1.2.0 > > > Users are confused how the SAVEPOINT ugprade mode works. > # It's not clear to users that the savepoint is drawn during the upgrade > process. > # It's not clear to users what happens when the process fails. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] gyfora merged pull request #365: [FLINK-29257] Clarify description of SAVEPOINT upgrade mode
gyfora merged PR #365: URL: https://github.com/apache/flink-kubernetes-operator/pull/365 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27511) Update operations playground for 1.15
[ https://issues.apache.org/jira/browse/FLINK-27511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27511: --- Labels: pull-request-available starter (was: starter) > Update operations playground for 1.15 > - > > Key: FLINK-27511 > URL: https://issues.apache.org/jira/browse/FLINK-27511 > Project: Flink > Issue Type: Sub-task > Components: Documentation / Training / Exercises >Reporter: David Anderson >Assignee: Roy Michael >Priority: Major > Labels: pull-request-available, starter > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-playgrounds] alpinegizmo merged pull request #34: [FLINK-27511][operations-playground] Update operations playground for 1.15
alpinegizmo merged PR #34: URL: https://github.com/apache/flink-playgrounds/pull/34 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a diff in pull request #20757: [FLINK-27919] Add FLIP-27-based source for data generation (FLIP-238)
stevenzwu commented on code in PR #20757:
URL: https://github.com/apache/flink/pull/20757#discussion_r974416546
##
flink-tests/src/test/java/org/apache/flink/api/connector/source/lib/util/RateLimitedSourceReaderITCase.java:
##
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.api.connector.source.lib.util;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.SourceReaderFactory;
+import org.apache.flink.api.connector.source.datagen.DataGeneratorSource;
+import org.apache.flink.api.connector.source.datagen.GeneratorFunction;
+import org.apache.flink.api.connector.source.lib.NumberSequenceSource;
+import org.apache.flink.runtime.testutils.MiniClusterResourceConfiguration;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.test.junit5.MiniClusterExtension;
+import org.apache.flink.util.TestLogger;
+
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CompletableFuture;
+import java.util.stream.Collectors;
+import java.util.stream.LongStream;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.assertj.core.api.Assertions.assertThat;
+
+/** An integration test for rate limiting built into the DataGeneratorSource.
*/
+public class RateLimitedSourceReaderITCase extends TestLogger {
Review Comment:
Can we add a unit test for the `GatedRateLimiter` with per checkpoint
quota/throttle? I guess the challenge is on how to validate the behavior. One
option is to have a map operator to calculate number of elements it received
per checkpoint cycle and emit an output as tuple .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #365: [FLINK-29257] Clarify description of SAVEPOINT upgrade mode
gyfora commented on code in PR #365: URL: https://github.com/apache/flink-kubernetes-operator/pull/365#discussion_r974378675 ## docs/content/docs/custom-resource/job-management.md: ## @@ -98,8 +98,8 @@ The `upgradeMode` setting controls both the stop and restore mechanisms as detai The three upgrade modes are intended to support different scenarios: 1. **stateless**: Stateless application upgrades from empty state - 2. **last-state**: Quick upgrades in any application state (even for failing jobs), does not require a healthy job as it always uses last checkpoint information. Manual recovery may be necessary if HA metadata is lost. - 3. **savepoint**: Use savepoint (when possible) for upgrade, providing maximal safety and possibility to serve as backup/fork point. + 2. **last-state**: Quick upgrades in any application state (even for failing jobs), does not require a healthy job as it always uses the latest checkpoint information. Manual recovery may be necessary if HA metadata is lost. + 3. **savepoint**: Use savepoint for upgrade, providing maximal safety and possibility to serve as backup/fork point. The savepoint will be created during the upgrade process. Note that the Flink job needs to be running to allow the savepoint to get created. If the job is in an unhealthy state, the last checkpoint will be used (if `kubernetes.operator.job.upgrade.last-state-fallback.enabled` is set to true). If the last checkpoint is not available, the job upgrade will fail. Review Comment: Technically it would be more precise to say: "unless `kubernetes.operator.job.upgrade.last-state-fallback.enabled` is set to false" because the default is true (you don't have to set it) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20856: [BP-1.16][FLINK-29025][docs] add docs for Hive dialect
flinkbot commented on PR #20856: URL: https://github.com/apache/flink/pull/20856#issuecomment-1251149181 ## CI report: * 2c0a744075586d6cac0e9c5043fc1960a2cee590 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong opened a new pull request, #20856: [BP-1.16][FLINK-29025][docs] add docs for Hive dialect
wuchong opened a new pull request, #20856: URL: https://github.com/apache/flink/pull/20856 This is a cherry-pick back to release-1.16 branch. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on a diff in pull request #20696: [FLINK-29025][docs] add docs for Hive dialect
wuchong commented on code in PR #20696:
URL: https://github.com/apache/flink/pull/20696#discussion_r974354501
##
docs/content/docs/dev/table/hiveCompatibility/hiveDialect/overview.md:
##
@@ -0,0 +1,97 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+aliases:
+- /dev/table/hiveCompatibility/hiveDialect/overview
+---
+
+
+# Hive Dialect
+
+Flink allows users to write SQL statements in Hive syntax when Hive dialect is
used.
+By providing compatibility with Hive syntax, we aim to improve the
interoperability with Hive and reduce the scenarios when users need to switch
between Flink and Hive in order to execute different statements.
+
+## Use Hive Dialect
+
+Flink currently supports two SQL dialects: `default` and `hive`. You need to
switch to Hive dialect
+before you can write in Hive syntax. The following describes how to set
dialect with
+SQL Client and Table API. Also notice that you can dynamically switch dialect
for each
+statement you execute. There's no need to restart a session to use a different
dialect.
+
+{{< hint warning >}}
+**Note:**
+
+- To use Hive dialect, you have to add dependencies related to Hive. Please
refer to [Hive dependencies]({{< ref "docs/connectors/table/hive/overview"
>}}#dependencies) for how to add the dependencies.
Review Comment:
Could you open another PR and issue for this improvement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a diff in pull request #20696: [FLINK-29025][docs] add docs for Hive dialect
wuchong commented on code in PR #20696:
URL: https://github.com/apache/flink/pull/20696#discussion_r974354109
##
docs/content/docs/dev/table/hiveCompatibility/hiveDialect/overview.md:
##
@@ -0,0 +1,97 @@
+---
+title: "Overview"
+weight: 1
+type: docs
+aliases:
+- /dev/table/hiveCompatibility/hiveDialect/overview
+---
+
+
+# Hive Dialect
+
+Flink allows users to write SQL statements in Hive syntax when Hive dialect is
used.
+By providing compatibility with Hive syntax, we aim to improve the
interoperability with Hive and reduce the scenarios when users need to switch
between Flink and Hive in order to execute different statements.
+
+## Use Hive Dialect
+
+Flink currently supports two SQL dialects: `default` and `hive`. You need to
switch to Hive dialect
+before you can write in Hive syntax. The following describes how to set
dialect with
+SQL Client and Table API. Also notice that you can dynamically switch dialect
for each
+statement you execute. There's no need to restart a session to use a different
dialect.
+
+{{< hint warning >}}
+**Note:**
+
+- To use Hive dialect, you have to add dependencies related to Hive. Please
refer to [Hive dependencies]({{< ref "docs/connectors/table/hive/overview"
>}}#dependencies) for how to add the dependencies.
Review Comment:
I think we should mention this in "Hive dependencies" because this is also
shared for using Hive dialect. You can add a note that the planner jar
switching is only needed when using Hive dialect and HiveServer2 Endpoint, but
this is the recommended setup for Hive integration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29313) Some config overrides are ignored when set under spec.flinkConfiguration
[
https://issues.apache.org/jira/browse/FLINK-29313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-29313:
---
Labels: pull-request-available (was: )
> Some config overrides are ignored when set under spec.flinkConfiguration
>
>
> Key: FLINK-29313
> URL: https://issues.apache.org/jira/browse/FLINK-29313
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.2.0
>Reporter: Matyas Orhidi
>Assignee: Gyula Fora
>Priority: Blocker
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.2.0
>
>
> Some
> [configs|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/configuration/#resourceuser-configuration]
> that can be specified under spec.flinkConfiguration won't take affect
> without an upgrade, e.g.:
> * {{kubernetes.operator.periodic.savepoint.interval}}
> * {{kubernetes.operator.savepoint.format.type}}
> These properties are used mainly from the so called 'observeConfig', and
> won't be available in the operator until the job is restarted. Ideally these
> should be changed without an upgrade, but at the moment they won't take
> affect at all.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29235) CVE-2022-25857 on flink-shaded
[ https://issues.apache.org/jira/browse/FLINK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606628#comment-17606628 ] Chesnay Schepler edited comment on FLINK-29235 at 9/19/22 2:50 PM: --- Mind you that the CVE likely doesn't really apply because we only use it to parse the configuration. was (Author: zentol): Mind you that the CVE likely doesn't apply because we only use it to parse the configuration. > CVE-2022-25857 on flink-shaded > -- > > Key: FLINK-29235 > URL: https://issues.apache.org/jira/browse/FLINK-29235 > Project: Flink > Issue Type: Bug > Components: BuildSystem / Shaded >Affects Versions: 1.15.2 >Reporter: Sergio Sainz >Assignee: Chesnay Schepler >Priority: Major > > flink-shaded-version uses snakeyaml v1.29 which is vulnerable to > CVE-2022-25857 > Ref: > https://nvd.nist.gov/vuln/detail/CVE-2022-25857 > https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-jackson/2.12.4-15.0/flink-shaded-jackson-2.12.4-15.0.pom > https://github.com/apache/flink-shaded/blob/master/flink-shaded-jackson-parent/flink-shaded-jackson-2/pom.xml#L73 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29235) CVE-2022-25857 on flink-shaded
[ https://issues.apache.org/jira/browse/FLINK-29235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17606628#comment-17606628 ] Chesnay Schepler commented on FLINK-29235: -- Mind you that the CVE likely doesn't apply because we only use it to parse the configuration. > CVE-2022-25857 on flink-shaded > -- > > Key: FLINK-29235 > URL: https://issues.apache.org/jira/browse/FLINK-29235 > Project: Flink > Issue Type: Bug > Components: BuildSystem / Shaded >Affects Versions: 1.15.2 >Reporter: Sergio Sainz >Assignee: Chesnay Schepler >Priority: Major > > flink-shaded-version uses snakeyaml v1.29 which is vulnerable to > CVE-2022-25857 > Ref: > https://nvd.nist.gov/vuln/detail/CVE-2022-25857 > https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-jackson/2.12.4-15.0/flink-shaded-jackson-2.12.4-15.0.pom > https://github.com/apache/flink-shaded/blob/master/flink-shaded-jackson-parent/flink-shaded-jackson-2/pom.xml#L73 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wuchong commented on pull request #20696: [FLINK-29025][docs] add docs for Hive dialect
wuchong commented on PR #20696:
URL: https://github.com/apache/flink/pull/20696#issuecomment-1251131456
LGTM. I added 4 commits to adjust the page order anduse dash-case
("hive-compatibility") instead of camelCase ("hiveComptibility") in URL and
remove "alias" front matter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on pull request #371: [FLINK-29313] Observe spec should include latest operator configs + improved test coverage
gyfora commented on PR #371: URL: https://github.com/apache/flink-kubernetes-operator/pull/371#issuecomment-1251131290 cc @morhidi -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
