[jira] [Commented] (FLINK-33734) Merge unaligned checkpoint state handle
[
https://issues.apache.org/jira/browse/FLINK-33734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795222#comment-17795222
]
Feifan Wang commented on FLINK-33734:
-
Thanks [~roman] , merge handle in JM in MetadataSerializer can only reduce the
size of metadata file, and dfs capacity tends not to be an issue. In the
Production example I mentioned above, the main problem is that checkpointing
takes too long. More specifically, we observed that it took more than 40
seconds for the JM to process the checkpoint ack rpc and more than 20 seconds
to serialize the metadata object. So I still think that handles should be
merged in TM.
{quote}1. Does it make sense to also merge state from multiple subtasks (as
implemented in FLINK-26803)?
{quote}
Yes, since multiple subtasks will reuse unaligned checkpoint files after the
ISSUE is completed, merging handles between multiple subtasks can further
reduce redundant data. But this may require changing the way the checkpoint
metadata objects are organized. And this optimization is constant level, but
merging handles within subtask can reduce the number of file paths from n^2 to
n. So I'm not sure if merging handles between subtasks is worth it at this
stage.
{quote}2. What happens when the delegate is in-memory state handle
(`ByteStreamStateHandle`)?
{quote}
IIUC, ByteStreamStateHandle in each
InputChannelStateHandle/ResultSubpartitionStateHandle is exclusive and uses a
random UUID as the handle name. I just looked at this code and saw that
FLINK-17972 was created while [~roman] were writing this code. I think the
MergedInputChannelStateHandle mentioned above is an implementation of
FLINK-17972. As for ByteStreamStateHandle, we may need to change channel
granular splitting to subtask granular splitting. WDYT [~roman] ?
> Merge unaligned checkpoint state handle
> ---
>
> Key: FLINK-33734
> URL: https://issues.apache.org/jira/browse/FLINK-33734
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Reporter: Feifan Wang
>Assignee: Feifan Wang
>Priority: Major
>
> h3. Background
> Unaligned checkpoint will write the inflight-data of all InputChannel and
> ResultSubpartition of the same subtask to the same file during checkpoint.
> The InputChannelStateHandle and ResultSubpartitionStateHandle organize the
> metadata of inflight-data at the channel granularity, which causes the file
> name to be repeated many times. When a job is under backpressure and task
> parallelism is high, the metadata of unaligned checkpoints will bloat. This
> will result in:
> # The amount of data reported by taskmanager to jobmanager increases, and
> jobmanager takes longer to process these RPC requests.
> # The metadata of the entire checkpoint becomes very large, and it takes
> longer to serialize and write it to dfs.
> Both of the above points ultimately lead to longer checkpoint duration.
> h3. A Production example
> Take our production job with a parallelism of 4800 as an example:
> # When there is no back pressure, checkpoint end-to-end duration is within 7
> seconds.
> # When under pressure: checkpoint end-to-end duration often exceeds 1
> minute. We found that jobmanager took more than 40 seconds to process rpc
> requests, and serialized metadata took more than 20 seconds.Some checkpoint
> statistics:
> |metadata file size|950 MB|
> |channel state count|12,229,854|
> |channel file count|5536|
> Of the 950MB in the metadata file, 68% are redundant file paths.
> We enabled log-based checkpoint on this job and hoped that the checkpoint
> could be completed within 30 seconds. This problem made it difficult to
> achieve this goal.
> h3. Propose changes
> I suggest introducing MergedInputChannelStateHandle and
> MergedResultSubpartitionStateHandle to eliminate redundant file paths.
> The taskmanager merges all InputChannelStateHandles with the same delegated
> StreamStateHandle in the same subtask into one MergedInputChannelStateHandle
> before reporting. When recovering from checkpoint, jobmangager converts
> MergedInputChannelStateHandle to InputChannelStateHandle collection before
> assigning state handle, and the rest of the process does not need to be
> changed.
> Structure of MergedInputChannelStateHandle :
>
> {code:java}
> { // MergedInputChannelStateHandle
> "delegate": {
> "filePath":
> "viewfs://hadoop-meituan/flink-yg15/checkpoints/retained/1234567/ab8d0c2f02a47586490b15e7a2c30555/chk-31/ffe54c0a-9b6e-4724-aae7-61b96bf8b1cf",
> "stateSize": 123456
> },
> "size": 2000,
> "subtaskIndex":0,
> "channels": [ // One InputChannel per element
> {
> "info": {
> "gateIdx": 0,
> "inputChannelIdx": 0
> },
>
Re: [PR] [doc-fix] - changed preference to settings for setting pycharm for pyflink ide setting section [flink]
Myasuka commented on code in PR #23903: URL: https://github.com/apache/flink/pull/23903#discussion_r1422036625 ## docs/content/docs/flinkDev/ide_setup.md: ## @@ -260,7 +260,7 @@ versions. [Flake8](https://pypi.org/project/flake8/) is used to enforce some coding guidelines. 1. Install flake8 for your Python interpreter using `pip install flake8`. -2. In PyCharm go to "Preferences" → "Tools" → "External Tools". +2. In PyCharm go to "Settings" → "Tools" → "External Tools". Review Comment: Could you please also update this in Chinese documents like [line 172](https://github.com/apache/flink/blob/master/docs/content.zh/docs/flinkDev/ide_setup.md?plain=1#L172) and [line 210](https://github.com/apache/flink/blob/master/docs/content.zh/docs/flinkDev/ide_setup.md?plain=1#L210)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17972) Consider restructuring channel state
[
https://issues.apache.org/jira/browse/FLINK-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Feifan Wang updated FLINK-17972:
Description:
(depends on rescaling for unaligned checkpoints (FLINK-17979))
Current structure is the following (this PR doesn't change it):
{code:java}
Each subtask reports to JM TaskStateSnapshot
each with zero ore more OperatorSubtaskState,
each with zero or more InputChannelStateHandle and
ResultSubpartitionStateHandle
each referencing an underlying StreamStateHandle
{code}
The underlying {{StreamStateHandle}} duplicates filename
({{{}ByteStreamStateHandle{}}} has it too at least because of
{{equals/hashcode}} I guess).
An alternative would be something like
{code:java}
Each subtask reports to JM TaskStateSnapshot
each with zero ore more OperatorSubtaskState
each with zero or one StreamStateHandle (for channel state)
each with zero or more InputChannelStateHandle and
ResultSubpartitionStateHandle{code}
{{{}(p{}}}{{{}robably, with StreamStateHandle{}}}{{ and InputChannelStateHandle
and ResultSubpartitionStateHandle}}{{ encapsulated)}}
It would be more effective (less data duplication) but probably also more
error-prone (implicit structure), less flexible (re-scaling).
(as discussed during introduction of {{StreamStateHandle.asBytesIfInMemory}}
[here|https://github.com/apache/flink/pull/12292#discussion_r429925802])
was:
(depends on rescaling for unaligned checkpoints (FLINK-17979))
Current structure is the following (this PR doesn't change it):
{code:java}
Each subtask reports to JM TaskStateSnapshot
each with zero ore more OperatorSubtaskState,
each with zero or more InputChannelStateHandle and
ResultSubpartitionStateHandle
each referencing an underlying StreamStateHandle
{code}
The underlying {{StreamStateHandle}} duplicates filename
({{ByteStreamStateHandle}} has it too at least because of {{equals/hashcode}} I
guess).
An alternative would be something like
{code:java}
Each subtask reports to JM TaskStateSnapshot
each with zero ore more OperatorSubtaskState
each with zero or one StreamStateHandle (for channel state)
each with zero or more InputChannelStateHandle and
ResultSubpartitionStateHandle{code}
{{(p}}{{robably, with StreamStateHandle}}{{ and InputChannelStateHandle and
ResultSubpartitionStateHandle}}{{ encapsulated)}}
It would be more effective (less data duplication) but probably also more
error-prone (implicit structure), less flexible (re-scaling).
(as discussed during introduction of {{StreamStateHandle.asBytesIfInMemory}}
[here|https://github.com/apache/flink/pull/12292#discussion_r429925802])
> Consider restructuring channel state
>
>
> Key: FLINK-17972
> URL: https://issues.apache.org/jira/browse/FLINK-17972
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.0
>Reporter: Roman Khachatryan
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned
>
> (depends on rescaling for unaligned checkpoints (FLINK-17979))
>
> Current structure is the following (this PR doesn't change it):
> {code:java}
> Each subtask reports to JM TaskStateSnapshot
> each with zero ore more OperatorSubtaskState,
> each with zero or more InputChannelStateHandle and
> ResultSubpartitionStateHandle
> each referencing an underlying StreamStateHandle
> {code}
> The underlying {{StreamStateHandle}} duplicates filename
> ({{{}ByteStreamStateHandle{}}} has it too at least because of
> {{equals/hashcode}} I guess).
> An alternative would be something like
> {code:java}
> Each subtask reports to JM TaskStateSnapshot
> each with zero ore more OperatorSubtaskState
> each with zero or one StreamStateHandle (for channel state)
> each with zero or more InputChannelStateHandle and
> ResultSubpartitionStateHandle{code}
> {{{}(p{}}}{{{}robably, with StreamStateHandle{}}}{{ and
> InputChannelStateHandle and ResultSubpartitionStateHandle}}{{ encapsulated)}}
>
> It would be more effective (less data duplication) but probably also more
> error-prone (implicit structure), less flexible (re-scaling).
> (as discussed during introduction of {{StreamStateHandle.asBytesIfInMemory}}
> [here|https://github.com/apache/flink/pull/12292#discussion_r429925802])
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] Bump org.apache.commons:commons-compress from 1.22 to 1.24.0 [flink-connector-pulsar]
dependabot[bot] opened a new pull request, #65: URL: https://github.com/apache/flink-connector-pulsar/pull/65 Bumps org.apache.commons:commons-compress from 1.22 to 1.24.0. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/flink-connector-pulsar/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28691) Improve cache hit rate of generated class
[ https://issues.apache.org/jira/browse/FLINK-28691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795194#comment-17795194 ] Dan Zou commented on FLINK-28691: - [~libenchao] Hi, I would like to take this ticket, could you please assign it to me? > Improve cache hit rate of generated class > - > > Key: FLINK-28691 > URL: https://issues.apache.org/jira/browse/FLINK-28691 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Dan Zou >Priority: Major > > In OLAP scenarios, compiling generated classes is very frequent, it will > consume a lot of CPU and large amount of generated classes will also takes up > a lot of space in metaspace, which will lead to frequent Full GC. > As we use a self-incrementing counter in CodeGenUtils#newName, it means we > could not get the same generated class between two queries even when they are > exactly the same. Maybe we could share the same generated class between > different queries if they has the same logic, it will be good for job latency > and resource consumption. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33792) Generate the same code for the same logic
[ https://issues.apache.org/jira/browse/FLINK-33792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795193#comment-17795193 ] Dan Zou commented on FLINK-33792: - [~libenchao] Hi, I would like to take this ticket, could you please assign it to me? > Generate the same code for the same logic > - > > Key: FLINK-33792 > URL: https://issues.apache.org/jira/browse/FLINK-33792 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Dan Zou >Priority: Major > > Generate the same code for the same logic, so that we may reuse the generated > code between different jobs. This is the precondition for FLINK-28691. The > current issue is we use a self-incrementing counter in CodeGenUtils#newName, > it means we could not get the same generated class between two queries even > when they are exactly the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33792) Generate the same code for the same logic
Dan Zou created FLINK-33792: --- Summary: Generate the same code for the same logic Key: FLINK-33792 URL: https://issues.apache.org/jira/browse/FLINK-33792 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Dan Zou Generate the same code for the same logic, so that we may reuse the generated code between different jobs. This is the precondition for FLINK-28691. The current issue is we use a self-incrementing counter in CodeGenUtils#newName, it means we could not get the same generated class between two queries even when they are exactly the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795171#comment-17795171 ] Yubin Li edited comment on FLINK-33780 at 12/11/23 6:14 AM: [~hackergin] Thanks for your attention:) In addition to the conceptual necessity (default catalog is also part of catalogs equaly and not differ from others), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production env using `CatalogStore`. !image-2023-12-11-14-14-10-002.png|width=821,height=576! was (Author: liyubin117): [~hackergin] Thanks for your attention:) In addition to the conceptual necessity (default catalog is also part of catalogs equaly and not differ from others), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production env using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Attachments: image-2023-12-11-13-47-29-623.png, > image-2023-12-11-14-14-10-002.png > > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795171#comment-17795171 ] Yubin Li edited comment on FLINK-33780 at 12/11/23 6:13 AM: [~hackergin] Thanks for your attention:) In addition to the conceptual necessity (default catalog is also part of catalogs equaly and not differ from others), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production env using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! was (Author: liyubin117): [~hackergin] Thanks for your attention:) In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production env using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Attachments: image-2023-12-11-13-47-29-623.png > > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795171#comment-17795171 ] Yubin Li edited comment on FLINK-33780 at 12/11/23 5:53 AM: [~hackergin] Thanks for your attention:) In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production env using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! was (Author: liyubin117): [~hackergin] In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production env using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Attachments: image-2023-12-11-13-47-29-623.png > > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795171#comment-17795171 ] Yubin Li edited comment on FLINK-33780 at 12/11/23 5:52 AM: [~hackergin] In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production env using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! was (Author: liyubin117): [~hackergin] In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production dev using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Attachments: image-2023-12-11-13-47-29-623.png > > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795171#comment-17795171 ] Yubin Li edited comment on FLINK-33780 at 12/11/23 5:51 AM: [~hackergin] In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore, I have implemented the statement on our production dev using `CatalogStore`. !image-2023-12-11-13-47-29-623.png|width=702,height=290! was (Author: liyubin117): [~hackergin] In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore. !image-2023-12-11-13-47-29-623.png|width=702,height=290! > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Attachments: image-2023-12-11-13-47-29-623.png > > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795170#comment-17795170 ] Gyula Fora commented on FLINK-33634: I don't think the idea was to migrate from status to condition. I think the status is completely fine as it is currently (of course there are always some improvements to be made later for new versions). Most of the status is not userfacing anyways. I think we can just take some part of it as you mentions like the job/jobmanager state and expose that. Things like save points / reconciliations succeeded I definitely would not add. > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795171#comment-17795171 ] Yubin Li commented on FLINK-33780: -- [~hackergin] In addition to the conceptual necessity (default catalog is also part of catalogs), I also found that the `show create catalog` statement proposed in https://issues.apache.org/jira/browse/FLINK-24939 would expose the absence of the default catalog in the CatalogStore. !image-2023-12-11-13-47-29-623.png|width=702,height=290! > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Attachments: image-2023-12-11-13-47-29-623.png > > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yubin Li updated FLINK-33780: - Attachment: image-2023-12-11-13-47-29-623.png > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Attachments: image-2023-12-11-13-47-29-623.png > > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33778][table] Cleanup usage of deprecated TableConfig#setIdleS… [flink]
liuyongvs commented on PR #23894: URL: https://github.com/apache/flink/pull/23894#issuecomment-1849338235 hi @snuyanzin @dawidwys could you help review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33779][table] Cleanup usage of deprecated BaseExpressions#cast [flink]
liuyongvs commented on PR #23895: URL: https://github.com/apache/flink/pull/23895#issuecomment-1849337918 hi @snuyanzin @dawidwys could you help review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33781][table] Cleanup usage of deprecated TableConfig#ctor [flink]
liuyongvs commented on PR #23897: URL: https://github.com/apache/flink/pull/23897#issuecomment-1849337647 hi @snuyanzin @dawidwys could you help review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33719][table] Cleanup the usage of deprecated StreamTableEnvir… [flink]
liuyongvs commented on PR #23898: URL: https://github.com/apache/flink/pull/23898#issuecomment-1849337372 hi @snuyanzin @dawidwys could you help review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32993][table] Datagen connector handles length-constrained fields according to the schema definition by default [flink]
liyubin117 commented on code in PR #23678:
URL: https://github.com/apache/flink/pull/23678#discussion_r1421917082
##
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/DataGenConnectorOptions.java:
##
@@ -118,10 +118,10 @@ public class DataGenConnectorOptions {
.withDescription("The proportion of null values.");
/** Placeholder {@link ConfigOption}. Not used for retrieving values. */
-public static final ConfigOption FIELD_VAR_LEN =
+public static final ConfigOption FIELD_VAR_LEN =
ConfigOptions.key(String.format("%s.#.%s", FIELDS, VAR_LEN))
-.floatType()
-.defaultValue(0f)
+.booleanType()
+.defaultValue(false)
Review Comment:
just fix typo, to make commits clear, I have divided the modifition into
another commit :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-32993][table] Datagen connector handles length-constrained fields according to the schema definition by default [flink]
liyubin117 commented on code in PR #23678:
URL: https://github.com/apache/flink/pull/23678#discussion_r1421917082
##
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/DataGenConnectorOptions.java:
##
@@ -118,10 +118,10 @@ public class DataGenConnectorOptions {
.withDescription("The proportion of null values.");
/** Placeholder {@link ConfigOption}. Not used for retrieving values. */
-public static final ConfigOption FIELD_VAR_LEN =
+public static final ConfigOption FIELD_VAR_LEN =
ConfigOptions.key(String.format("%s.#.%s", FIELDS, VAR_LEN))
-.floatType()
-.defaultValue(0f)
+.booleanType()
+.defaultValue(false)
Review Comment:
just to fix typo, to make commits clear, I have divided the modifition into
another commit :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795152#comment-17795152
]
luoyuxia edited comment on FLINK-33788 at 12/11/23 3:43 AM:
[~shaoguang] Cool! Thanks. One hint is that we may only need to consider the
caller of
{code:java}
String getUnescapedName(
HiveParserASTNode tableOrColumnNode,
@Nullable String currentCatalog,
@Nullable String currentDatabase) {
xx
} {code}
Since it's the only entrypoint to pass catalog. For database name and table
name, it's safe and a must to convert it to lowercase since Hive is
case-insentive. But we shouldn't convert catalog name to lowercase.
was (Author: luoyuxia):
[~shaoguang] Cool! Thanks. One hint is that we may only need to consider the
caller of
{code:java}
String getUnescapedName(
HiveParserASTNode tableOrColumnNode,
@Nullable String currentCatalog,
@Nullable String currentDatabase) {
xx
} {code}
Since it's the only entrypoint to pass catalog. For database name and table
name, it's safe and a must to convert it to lowercase since Hive is
case-insentive. But we shouldn't convert catalog name to lowercase.
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
luoyuxia reassigned FLINK-33788:
Assignee: lishaoguang
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Assignee: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795152#comment-17795152
]
luoyuxia edited comment on FLINK-33788 at 12/11/23 3:42 AM:
[~shaoguang] Cool! Thanks. One hint is that we may only need to consider the
caller of
{code:java}
String getUnescapedName(
HiveParserASTNode tableOrColumnNode,
@Nullable String currentCatalog,
@Nullable String currentDatabase) {
xx
} {code}
Since it's the only entrypoint to pass catalog. For database name and table
name, it's safe and a must to convert it to lowercase since Hive is
case-insentive. But we shouldn't convert catalog name to lowercase.
was (Author: luoyuxia):
[~shaoguang] Cool! Thanks. One hint is that we may only need to consider the
caller of
{code:java}
String getUnescapedName(
HiveParserASTNode tableOrColumnNode,
@Nullable String currentCatalog,
@Nullable String currentDatabase) {
xx
} {code}
Since it's the only entrypoint to pass catalog. For database name and table
name, it's safe and a must to convert it to lowercase since Hive is
case-insentive. But we shouldn't convert catalog name to lowercase.
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795152#comment-17795152
]
luoyuxia commented on FLINK-33788:
--
[~shaoguang] Cool! Thanks. One hint is that we may only need to consider the
caller of
{code:java}
String getUnescapedName(
HiveParserASTNode tableOrColumnNode,
@Nullable String currentCatalog,
@Nullable String currentDatabase) {
xx
} {code}
Since it's the only entrypoint to pass catalog. For database name and table
name, it's safe and a must to convert it to lowercase since Hive is
case-insentive. But we shouldn't convert catalog name to lowercase.
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33780) Support to store default catalog in CatalogStore
[ https://issues.apache.org/jira/browse/FLINK-33780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795151#comment-17795151 ] Feng Jin commented on FLINK-33780: -- Hi, [~liyubin117] Thanks for pointing this . could you provide a more detailed description of what problems may arise if we don't save the default catalog ? > Support to store default catalog in CatalogStore > > > Key: FLINK-33780 > URL: https://issues.apache.org/jira/browse/FLINK-33780 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > > Flink initially creates a default catalog which is included in the > `Map catalogs`, but is not stored in the CatalogStore. > After conducting thorough investigation, I've determined that the necessary > modification can be made within the `CatalogManager`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33784) CatalogStoreFactory can not be configured via StreamExecutionEnvironment
[ https://issues.apache.org/jira/browse/FLINK-33784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795149#comment-17795149 ] Feng Jin commented on FLINK-33784: -- [~twalthr] Thank you for pointing out this issue, I would like fix it, please assign this task to me. Indeed, some modifications need to be made to handle the `ReadableConfig` for the current logic. How about introduce a DelegatingReadableConfig , so we can deal with prefix options. > CatalogStoreFactory can not be configured via StreamExecutionEnvironment > > > Key: FLINK-33784 > URL: https://issues.apache.org/jira/browse/FLINK-33784 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Timo Walther >Priority: Major > > The logic in TableEnvironment.create() has well-defined ordering which allows > to configure most settings via StreamExecutionEnvironment and > flink-conf.yaml. The discovery of CatalogStoreFactory should be postponed > until the final configuration is merged. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32993][table] Datagen connector handles length-constrained fields according to the schema definition by default [flink]
liyubin117 commented on code in PR #23678:
URL: https://github.com/apache/flink/pull/23678#discussion_r1421910481
##
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/RandomGeneratorVisitor.java:
##
@@ -489,6 +465,16 @@ protected DataGeneratorContainer defaultMethod(LogicalType
logicalType) {
throw new ValidationException("Unsupported type: " + logicalType);
}
+private ConfigOption getLengthOption(Supplier
defaultLengthSupplier) {
Review Comment:
wise idea !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-33791) Fix NPE when array is null in PostgresArrayConverter in flink-connector-jdbc
JingWei Li created FLINK-33791:
--
Summary: Fix NPE when array is null in PostgresArrayConverter in
flink-connector-jdbc
Key: FLINK-33791
URL: https://issues.apache.org/jira/browse/FLINK-33791
Project: Flink
Issue Type: Bug
Components: Connectors / JDBC
Reporter: JingWei Li
{code:java}
// private JdbcDeserializationConverter createPostgresArrayConverter(ArrayType
arrayType) {
// Since PGJDBC 42.2.15 (https://github.com/pgjdbc/pgjdbc/pull/1194)
bytea[] is wrapped in
// primitive byte arrays
final Class elementClass =
LogicalTypeUtils.toInternalConversionClass(arrayType.getElementType());
final JdbcDeserializationConverter elementConverter =
createNullableInternalConverter(arrayType.getElementType());
return val -> {
@SuppressWarnings("unchecked")
T pgArray = (T) val;
Object[] in = (Object[]) pgArray.getArray();
final Object[] array = (Object[]) Array.newInstance(elementClass,
in.length);
for (int i = 0; i < in.length; i++) {
array[i] = elementConverter.deserialize(in[i]);
}
return new GenericArrayData(array);
};
} {code}
When use this method, array is null pgArray.getArray() will throw NPE。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33790) Upsert statement filter unique key field colume in mysql dielact
JingWei Li created FLINK-33790:
--
Summary: Upsert statement filter unique key field colume in mysql
dielact
Key: FLINK-33790
URL: https://issues.apache.org/jira/browse/FLINK-33790
Project: Flink
Issue Type: Improvement
Components: Connectors / JDBC
Reporter: JingWei Li
example: `col2` and `col4` is unique key in table `my_table`
{code:java}
INSERT INTO `my_table`(`col1`, `col2`, `col3`, `col4`, `col5`)
VALUES (?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
`col1`=VALUES(`col1`),
`col2`=VALUES(`col2`),
`col3`=VALUES(`col3`),
`col4`=VALUES(`col4`),
`col5`=VALUES(`col5`){code}
result:
{code:java}
INSERT INTO `my_table`(`col1`, `col2`, `col3`, `col4`, `col5`)
VALUES (?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
`col1`=VALUES(`col1`),
`col3`=VALUES(`col3`),
`col5`=VALUES(`col5`) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-32993][table] Datagen connector handles length-constrained fields according to the schema definition by default [flink]
LadyForest commented on code in PR #23678:
URL: https://github.com/apache/flink/pull/23678#discussion_r1421891193
##
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/DataGenConnectorOptions.java:
##
@@ -118,10 +118,10 @@ public class DataGenConnectorOptions {
.withDescription("The proportion of null values.");
/** Placeholder {@link ConfigOption}. Not used for retrieving values. */
-public static final ConfigOption FIELD_VAR_LEN =
+public static final ConfigOption FIELD_VAR_LEN =
ConfigOptions.key(String.format("%s.#.%s", FIELDS, VAR_LEN))
-.floatType()
-.defaultValue(0f)
+.booleanType()
+.defaultValue(false)
Review Comment:
I'm a little bit confused. Why change this config type?
##
flink-table/flink-table-api-java-bridge/src/main/java/org/apache/flink/connector/datagen/table/RandomGeneratorVisitor.java:
##
@@ -489,6 +465,16 @@ protected DataGeneratorContainer defaultMethod(LogicalType
logicalType) {
throw new ValidationException("Unsupported type: " + logicalType);
}
+private ConfigOption getLengthOption(Supplier
defaultLengthSupplier) {
Review Comment:
`ArrayType`, `MultisetType,` and `MapType` can also use this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-32881][checkpoint] Support triggering savepoint in detach mode for CLI and dumping all pending savepoint ids by rest api [flink]
Zakelly commented on PR #23253: URL: https://github.com/apache/flink/pull/23253#issuecomment-1849238533 > > Hi @xiangforever2014 @masteryhx sorry for jumping in... And I'm wondering if we could use the capital '-D' for short option? Or I suggest we do not leave any short option for this, only using '-detach'. WDYT? > > And I agree we could adjust all these CLI options in 2.0. > > Thanks for your comment~ But I wonder that is there any problem in current implementation? Or why we need to change it to "-D" or "-detach"? Hope to get your suggestion~ Hi @xiangforever2014 . Short commands are designed to be readable or easy for users to remember. It can be omitted if there is no suitable abbreviation. The '-dcp' is not that easy to remember or understand, user may not know what it stands for. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [DRAFT][FLINK-32667] Use standalone store and embedding writer for jobs with no-restart-strategy in session cluster [flink]
flinkbot commented on PR #23904: URL: https://github.com/apache/flink/pull/23904#issuecomment-1849236984 ## CI report: * 81eb39d1ed6d0129ca7548051210100af961ed32 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32738][formats] PROTOBUF format supports projection push down [flink]
ljw-hit commented on PR #23323: URL: https://github.com/apache/flink/pull/23323#issuecomment-1849235718 @zhougit86 Thank you for your patience and contribution. I will complete the review within the next two weeks and provide comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [DRAFT][FLINK-32667] Use standalone store and embedding writer for jobs with no-restart-strategy in session cluster [flink]
KarmaGYZ opened a new pull request, #23904: URL: https://github.com/apache/flink/pull/23904 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795144#comment-17795144
]
lishaoguang edited comment on FLINK-33788 at 12/11/23 2:32 AM:
---
[~luoyuxia] I will try to fix it.But I found there are 11 lowercase convert
conversions,I should test theme one by one.
was (Author: shaoguang):
[~luoyuxia] I will try to fix it.But I find there are 11 lowercase convert
conversions,I should test theme one by one.
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795144#comment-17795144
]
lishaoguang commented on FLINK-33788:
-
[~luoyuxia] I will try to fix it.But I find there are 11 lowercase convert
conversions,I should test theme one by one.
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33741) Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic related configuration
[
https://issues.apache.org/jira/browse/FLINK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795142#comment-17795142
]
Yun Tang commented on FLINK-33741:
--
[~zhoujira86] I think there exists valuable information in the RocksDB
statistics, assigned to you, please go ahead.
> Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic
> related configuration
> ---
>
> Key: FLINK-33741
> URL: https://issues.apache.org/jira/browse/FLINK-33741
> Project: Flink
> Issue Type: New Feature
>Reporter: xiaogang zhou
>Assignee: xiaogang zhou
>Priority: Major
>
> I think we can also parse the multi-line string of the rocksdb statistics.
> {code:java}
> // code placeholder
> /**
> * DB implements can export properties about their state
> * via this method on a per column family level.
> *
> * If {@code property} is a valid property understood by this DB
> * implementation, fills {@code value} with its current value and
> * returns true. Otherwise returns false.
> *
> * Valid property names include:
> *
> * "rocksdb.num-files-at-levelN" - return the number of files at
> * level N, where N is an ASCII representation of a level
> * number (e.g. "0").
> * "rocksdb.stats" - returns a multi-line string that describes statistics
> * about the internal operation of the DB.
> * "rocksdb.sstables" - returns a multi-line string that describes all
> *of the sstables that make up the db contents.
> *
> *
> * @param columnFamilyHandle {@link org.rocksdb.ColumnFamilyHandle}
> * instance, or null for the default column family.
> * @param property to be fetched. See above for examples
> * @return property value
> *
> * @throws RocksDBException thrown if error happens in underlying
> *native library.
> */
> public String getProperty(
> /* @Nullable */ final ColumnFamilyHandle columnFamilyHandle,
> final String property) throws RocksDBException { {code}
>
> Then we can directly export these rt latency number in metrics.
>
> I'd like to introduce 2 rocksdb statistic related configuration.
> Then we can customize stats
> {code:java}
> // code placeholder
> Statistics s = new Statistics();
> s.setStatsLevel(EXCEPT_TIME_FOR_MUTEX);
> currentOptions.setStatsDumpPeriodSec(internalGetOption(RocksDBConfigurableOptions.STATISTIC_DUMP_PERIOD))
> .setStatistics(s); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33741) Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic related configuration
[
https://issues.apache.org/jira/browse/FLINK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Tang reassigned FLINK-33741:
Assignee: xiaogang zhou
> Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic
> related configuration
> ---
>
> Key: FLINK-33741
> URL: https://issues.apache.org/jira/browse/FLINK-33741
> Project: Flink
> Issue Type: New Feature
>Reporter: xiaogang zhou
>Assignee: xiaogang zhou
>Priority: Major
>
> I think we can also parse the multi-line string of the rocksdb statistics.
> {code:java}
> // code placeholder
> /**
> * DB implements can export properties about their state
> * via this method on a per column family level.
> *
> * If {@code property} is a valid property understood by this DB
> * implementation, fills {@code value} with its current value and
> * returns true. Otherwise returns false.
> *
> * Valid property names include:
> *
> * "rocksdb.num-files-at-levelN" - return the number of files at
> * level N, where N is an ASCII representation of a level
> * number (e.g. "0").
> * "rocksdb.stats" - returns a multi-line string that describes statistics
> * about the internal operation of the DB.
> * "rocksdb.sstables" - returns a multi-line string that describes all
> *of the sstables that make up the db contents.
> *
> *
> * @param columnFamilyHandle {@link org.rocksdb.ColumnFamilyHandle}
> * instance, or null for the default column family.
> * @param property to be fetched. See above for examples
> * @return property value
> *
> * @throws RocksDBException thrown if error happens in underlying
> *native library.
> */
> public String getProperty(
> /* @Nullable */ final ColumnFamilyHandle columnFamilyHandle,
> final String property) throws RocksDBException { {code}
>
> Then we can directly export these rt latency number in metrics.
>
> I'd like to introduce 2 rocksdb statistic related configuration.
> Then we can customize stats
> {code:java}
> // code placeholder
> Statistics s = new Statistics();
> s.setStatsLevel(EXCEPT_TIME_FOR_MUTEX);
> currentOptions.setStatsDumpPeriodSec(internalGetOption(RocksDBConfigurableOptions.STATISTIC_DUMP_PERIOD))
> .setStatistics(s); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33788) Table not found for catalogname is capital
[
https://issues.apache.org/jira/browse/FLINK-33788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795137#comment-17795137
]
luoyuxia commented on FLINK-33788:
--
[~shaoguang] Thanks for reporting it. As Hive is case-insentive, when using
Hive dialect, we'll always convert table name, database name, catalog name to
lowercase. But it seems we shouldn't convert catalog name to lowercase
directly, otherwise, it won't find the catalog reigsterd in flink. I think it's
a bug that should be fixed. Could you please help fix if you're free? I think
1.17, 1.18, master should be fixed.
But for master, you should submit a pr in a dedicated repo for Hive.
https://github.com/apache/flink-connector-hive
> Table not found for catalogname is capital
> --
>
> Key: FLINK-33788
> URL: https://issues.apache.org/jira/browse/FLINK-33788
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.16.1
>Reporter: lishaoguang
>Priority: Major
> Attachments: image-2023-12-09-20-26-41-150.png, screenshot-1.png
>
>
> {code:java}
> String runMode=args[0];
> String sql = "select * from student";
> String sql = "insert into table student select * from student;";
> String sql = args[1];
> String catalogName = "myHive";
> String defaultDatabase = args[2];
> RuntimeExecutionMode executionMode = "batch".equals(runMode) ?
> RuntimeExecutionMode.BATCH : RuntimeExecutionMode.STREAMING;
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setRuntimeMode(executionMode);
> StreamTableEnvironment tenv = StreamTableEnvironment.create(env);
>
> tenv.getConfig().setSqlDialect(SqlDialect.HIVE);
> Configuration conf = new Configuration();
> conf.addResource("/ect/hadoop/conf");
> conf.addResource("/ect/hive/conf");
> HiveConf hiveConf = new HiveConf(conf, Configuration.class);
> String hiveVersion = "3.1.3";
> HiveCatalog hiveCatalog = new HiveCatalog(catalogName,
> defaultDatabase, hiveConf, hiveVersion, true);
> tenv.registerCatalog(catalogName, hiveCatalog);
> tenv.useCatalog(catalogName);
> tenv.useDatabase(defaultDatabase);
> tenv.executeSql(sql).print(); {code}
> Even if the table exists in hive, the query will still report an error
> like"Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:14
> Table not found 'student'"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795126#comment-17795126 ] Ryan van Huuksloot edited comment on FLINK-33634 at 12/10/23 11:00 PM: --- Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? It would lead to a duplication of logic where there is a Condition with `type:Ready` and having `JobStatus` in the `.status.jobStatus`. *Note: I think `type: Ready` is too vague for Flink.* I'm not an operator expert but is it common to duplicate with objects and Conditions in status? If yes, (y) you can ignore the rest of the post. If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? (There is a lot of usages in the operator of `.getStatus().getJobStatus().getState()`) Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous and we could duplicate the implementation in the short term. Potentially moving towards a spec for FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. Apologies if you got spam emailed. It kept submitting when trying to do same paragraph new lines. was (Author: JIRAUSER285640): Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? It would lead to a duplication of logic where there is a Condition with `type:Ready` and having `JobStatus` in the `.status.jobStatus`. *Note: I think `type: Ready` is too vague for Flink.* I'm not an operator expert but is it common to duplicate with objects and Conditions in status? If yes, (y) you can ignore the rest of the post. If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? (There is a lot of usages in the operator of `.getStatus().getJobStatus().getState()`) Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous and we could duplicate the implementation in the short term. Potentially moving towards a spec for FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795126#comment-17795126 ] Ryan van Huuksloot edited comment on FLINK-33634 at 12/10/23 10:59 PM: --- Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? It would lead to a duplication of logic where there is a Condition with `type:Ready` and having `JobStatus` in the `.status.jobStatus`. *Note: I think `type: Ready` is too vague for Flink.* I'm not an operator expert but is it common to duplicate with objects and Conditions in status? If yes, (y) you can ignore the rest of the post. If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? (There is a lot of usages in the operator of `.getStatus().getJobStatus().getState()`) Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous and we could duplicate the implementation in the short term. Potentially moving towards a spec for FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. was (Author: JIRAUSER285640): Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? It would lead to a duplication of logic where there is a Condition with `type:Ready` and having `JobStatus` in the `.status.jobStatus`. Note: I think `type: Ready` is too vague for Flink. I'm not an operator expert but is it common to duplicate with objects and Conditions in status? If yes, (y) you can ignore the rest of the post. If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? (There is a lot of usages in the operator of `.getStatus().getJobStatus().getState()`) Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous and we could duplicate the implementation in the short term. Potentially moving towards a spec for FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795126#comment-17795126 ] Ryan van Huuksloot edited comment on FLINK-33634 at 12/10/23 10:59 PM: --- Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? It would lead to a duplication of logic where there is a Condition with `type:Ready` and having `JobStatus` in the `.status.jobStatus`. Note: I think `type: Ready` is too vague for Flink. I'm not an operator expert but is it common to duplicate with objects and Conditions in status? If yes, (y) you can ignore the rest of the post. If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? (There is a lot of usages in the operator of `.getStatus().getJobStatus().getState()`) Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous and we could duplicate the implementation in the short term. Potentially moving towards a spec for FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. was (Author: JIRAUSER285640): Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? It would lead to a duplication of logic where there is a Condition with `type:Ready` and having `JobStatus` in the `.status.jobStatus`. Note: I think `type: Ready` is too vague for Flink. I'm not an operator expert but is it common to duplicate with objects and Conditions in status? If yes, (y) you can ignore the rest of the post. If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? (There is a lot of usages in the operator of `.getStatus().getJobStatus().getState()`) Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous and we could duplicate the implementation in the short term. Potentially moving towards a spec for FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795126#comment-17795126 ] Ryan van Huuksloot edited comment on FLINK-33634 at 12/10/23 10:57 PM: --- Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? It would lead to a duplication of logic where there is a Condition with `type:Ready` and having `JobStatus` in the `.status.jobStatus`. Note: I think `type: Ready` is too vague for Flink. I'm not an operator expert but is it common to duplicate with objects and Conditions in status? If yes, (y) you can ignore the rest of the post. If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? (There is a lot of usages in the operator of `.getStatus().getJobStatus().getState()`) Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous and we could duplicate the implementation in the short term. Potentially moving towards a spec for FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. was (Author: JIRAUSER285640): Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? I think the Condition K8 spec could be a good migration - the caveat being that it is more annoying to search through a list for something like `.getStatus().getJobStatus().getState()` which is used throughout the operator. That being said, I think duplication of the logic would lead to a weird situation where there is a Condition with type:Ready and having `JobStatus` in the `.status.jobStatus`? I'm not an operator expert but is this common to duplicate? If yes, (y). If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous Thoughts on starting with what currently exists in the `jobManagerDeploymentStatus`. It might make more sense as a condition than a separate status object. There might be other custom status objects that could also be standardized eventually. The caveat being the `jobManagerDeploymentStatus` is used Potentially moving towards a spec for the FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready
[jira] [Comment Edited] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795126#comment-17795126 ] Ryan van Huuksloot edited comment on FLINK-33634 at 12/10/23 10:47 PM: --- Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? I think the Condition K8 spec could be a good migration - the caveat being that it is more annoying to search through a list for something like `.getStatus().getJobStatus().getState()` which is used throughout the operator. That being said, I think duplication of the logic would lead to a weird situation where there is a Condition with type:Ready and having `JobStatus` in the `.status.jobStatus`? I'm not an operator expert but is this common to duplicate? If yes, (y). If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous Thoughts on starting with what currently exists in the `jobManagerDeploymentStatus`. It might make more sense as a condition than a separate status object. There might be other custom status objects that could also be standardized eventually. The caveat being the `jobManagerDeploymentStatus` is used Potentially moving towards a spec for the FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. was (Author: JIRAUSER285640): Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? I think the Condition K8 spec could be a good migration - the caveat being that it is more annoying to search through a list for something like `.getStatus().getJobStatus().getState()` which is used throughout the operator. That being said, I think duplication of the logic would lead to a weird situation where there is a Condition with type:Ready and having `JobStatus` in the `.status.jobStatus`? I'm not an operator expert but is this common to duplicate? If yes, (y). If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous Thoughts on starting with what currently exists in the `jobManagerDeploymentStatus`. It might make more sense as a condition than a separate status object. There might be other custom status objects that could also be standardized eventually. The caveat being the `jobManagerDeploymentStatus` is used Potentially moving towards a spec for the FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in
[jira] [Commented] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795126#comment-17795126 ] Ryan van Huuksloot commented on FLINK-33634: Is the idea of this issue to migrate the current custom `Status` objects to a more uniform and standardized spec? I think the Condition K8 spec could be a good migration - the caveat being that it is more annoying to search through a list for something like `.getStatus().getJobStatus().getState()` which is used throughout the operator. That being said, I think duplication of the logic would lead to a weird situation where there is a Condition with type:Ready and having `JobStatus` in the `.status.jobStatus`? I'm not an operator expert but is this common to duplicate? If yes, (y). If no, any smart idea how to make it less annoying to consistently search through all of the Conditions in the operator code? Maybe it is easiest to start with something like `jobManagerDeploymentStatus`. It seems pretty innocuous Thoughts on starting with what currently exists in the `jobManagerDeploymentStatus`. It might make more sense as a condition than a separate status object. There might be other custom status objects that could also be standardized eventually. The caveat being the `jobManagerDeploymentStatus` is used Potentially moving towards a spec for the FlinkDeployment like: ``` status: conditions: - type: JobManagerReady status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:39:51Z' observedGeneration: 1 reason: Ready message: "JobManager is running and ready to receive REST API calls." - type: ReconciliationSucceeded status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' reason: ReconciliationSucceeded message: "Reconciliation succeeded." - type: JobRunning status: "True" lastTransitionTime: '2023-11-23T12:38:51Z' lastHeartbeatTime: '2023-11-23T12:40:51Z' observedGeneration: 12 reason: Running message: "Job is running." jobId: 0b0f0c0a-0b0f-0c0a-0b0f-0c0a0b0f0c0a jobName: flink-kubernetes-operator-test ``` Could add information about the Savepoint/Checkpoint as well. > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634 ] Ryan van Huuksloot deleted comment on FLINK-33634: was (Author: JIRAUSER285640): Thoughts on converting the current Status representations to Conditions prior to adding more Conditions? We can then deprecate the current Statuses. From my perspective the Statuses that exist today is the Flink Operator's version of Conditions and it would be good to centralize/standardize them. Example (Flink Deployment): status: conditions: - type: JobManagerReady lastTransitionTime: '2023-11-23T12:38:51Z' status > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795121#comment-17795121 ] Ryan van Huuksloot commented on FLINK-33634: Thoughts on converting the current Status representations to Conditions prior to adding more Conditions? We can then deprecate the current Statuses. From my perspective the Statuses that exist today is the Flink Operator's version of Conditions and it would be good to centralize/standardize them. Example (Flink Deployment): status: conditions: - type: JobManagerReady lastTransitionTime: '2023-11-23T12:38:51Z' status > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634 ] Ryan van Huuksloot deleted comment on FLINK-33634: was (Author: JIRAUSER285640): Thoughts on converting the current Status representations to Conditions prior to adding more Conditions? We can then deprecate the current Statuses. From my perspective the Statuses that exist today is the Flink Operator's version of Conditions and it would be good to centralize/standardize them. Example (Flink Deployment): status: conditions: - type: JobManager lastTransitionTime: '2023-11-23T12:38:51Z' > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33634) Add Conditions to Flink CRD's Status field
[ https://issues.apache.org/jira/browse/FLINK-33634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795120#comment-17795120 ] Ryan van Huuksloot commented on FLINK-33634: Thoughts on converting the current Status representations to Conditions prior to adding more Conditions? We can then deprecate the current Statuses. From my perspective the Statuses that exist today is the Flink Operator's version of Conditions and it would be good to centralize/standardize them. Example (Flink Deployment): status: conditions: - type: JobManager lastTransitionTime: '2023-11-23T12:38:51Z' > Add Conditions to Flink CRD's Status field > -- > > Key: FLINK-33634 > URL: https://issues.apache.org/jira/browse/FLINK-33634 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.7.0 >Reporter: Tony Garrard >Priority: Major > > From > [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties] > it is considered best practice to provide Conditions in the Status of CRD's. > Some tooling even expects there to be a Conditions field in the status of a > CR. This issue to to propose adding a Conditions field to the CR status > e.g. > status: > conditions: > - lastTransitionTime: '2023-11-23T12:38:51Z' > status: 'True' > type: Ready -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33789) Expose restart time as a metric
[ https://issues.apache.org/jira/browse/FLINK-33789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33789: -- Summary: Expose restart time as a metric (was: Expose restart time as metric) > Expose restart time as a metric > --- > > Key: FLINK-33789 > URL: https://issues.apache.org/jira/browse/FLINK-33789 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.8.0 > > > FLINK-30593 added restart time tracking. It would be convenient to also > report is as a metric. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33789) Expose restart time as metric
[ https://issues.apache.org/jira/browse/FLINK-33789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33789: -- Description: FLINK-30593 added restart time tracking. It would be convenient to also report is as a metric. (was: FLINK-30593 added restart time tracking. It would be convenient to also report is as metric.) > Expose restart time as metric > - > > Key: FLINK-33789 > URL: https://issues.apache.org/jira/browse/FLINK-33789 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.8.0 > > > FLINK-30593 added restart time tracking. It would be convenient to also > report is as a metric. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33789) Expose restart time as metric
[ https://issues.apache.org/jira/browse/FLINK-33789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Fedulov updated FLINK-33789: -- Description: FLINK-30593 added restart time tracking. It would be convenient to also report is as metric. (was: Currently the autoscaler uses a preconfigured restart time for the job. We should dynamically adjust this on the observered restart times for scale operations.) > Expose restart time as metric > - > > Key: FLINK-33789 > URL: https://issues.apache.org/jira/browse/FLINK-33789 > Project: Flink > Issue Type: New Feature > Components: Autoscaler, Kubernetes Operator >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.8.0 > > > FLINK-30593 added restart time tracking. It would be convenient to also > report is as metric. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33789) Expose restart time as metric
Alexander Fedulov created FLINK-33789: - Summary: Expose restart time as metric Key: FLINK-33789 URL: https://issues.apache.org/jira/browse/FLINK-33789 Project: Flink Issue Type: New Feature Components: Autoscaler, Kubernetes Operator Reporter: Alexander Fedulov Assignee: Alexander Fedulov Fix For: kubernetes-operator-1.8.0 Currently the autoscaler uses a preconfigured restart time for the job. We should dynamically adjust this on the observered restart times for scale operations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [doc-fix] - changed preference to settings for setting pycharm for pyflink ide setting section [flink]
flinkbot commented on PR #23903: URL: https://github.com/apache/flink/pull/23903#issuecomment-1849028559 ## CI report: * 4366e0adc981baefb7ca51e67799fd86317804a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [doc-fix] - changed preference to settings for setting pycharm for pyflink ide setting section [flink]
adipolak opened a new pull request, #23903: URL: https://github.com/apache/flink/pull/23903 ## What is the purpose of the change setting pycharm for pyflink project - fixing from going to reference (which dosn't exist) to settings ## Brief change log - doc text update - changed preference to settings ## Verifying this change no test required for this ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? ( no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33536] Fix Flink Table API CSV streaming sink fails with IOException: Stream closed [flink]
PrabhuJoseph commented on PR #23725: URL: https://github.com/apache/flink/pull/23725#issuecomment-1849009696 @Samrat002 I have verified the patch with filesystem connector using both csv and json format. The patch is working fine. LGTM, +1 (non-binding) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Fix abnormal prompt information: When sorting in non-time attribute f… [flink]
flinkbot commented on PR #23902: URL: https://github.com/apache/flink/pull/23902#issuecomment-1848984657 ## CI report: * 29f82bd2a211fc7dee9f636b7d11aed091e3c5b3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Fix abnormal prompt information: When sorting in non-time attribute f… [flink]
Gianzie opened a new pull request, #23902: URL: https://github.com/apache/flink/pull/23902 …ields ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33741) Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic related configuration
[
https://issues.apache.org/jira/browse/FLINK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaogang zhou updated FLINK-33741:
--
Summary: Exposed Rocksdb statistics in Flink metrics and introduce 2
Rocksdb statistic related configuration (was: Introduce stat dump period and
statsLevel configuration)
> Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic
> related configuration
> ---
>
> Key: FLINK-33741
> URL: https://issues.apache.org/jira/browse/FLINK-33741
> Project: Flink
> Issue Type: New Feature
>Reporter: xiaogang zhou
>Priority: Major
>
> I'd like to introduce 2 rocksdb statistic related configuration.
> Then we can customize stats
> {code:java}
> // code placeholder
> Statistics s = new Statistics();
> s.setStatsLevel(EXCEPT_TIME_FOR_MUTEX);
> currentOptions.setStatsDumpPeriodSec(internalGetOption(RocksDBConfigurableOptions.STATISTIC_DUMP_PERIOD))
> .setStatistics(s); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33741) Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic related configuration
[
https://issues.apache.org/jira/browse/FLINK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaogang zhou updated FLINK-33741:
--
Description:
I think we can also parse the multi-line string of the rocksdb statistics.
{code:java}
// code placeholder
/**
* DB implements can export properties about their state
* via this method on a per column family level.
*
* If {@code property} is a valid property understood by this DB
* implementation, fills {@code value} with its current value and
* returns true. Otherwise returns false.
*
* Valid property names include:
*
* "rocksdb.num-files-at-levelN" - return the number of files at
* level N, where N is an ASCII representation of a level
* number (e.g. "0").
* "rocksdb.stats" - returns a multi-line string that describes statistics
* about the internal operation of the DB.
* "rocksdb.sstables" - returns a multi-line string that describes all
*of the sstables that make up the db contents.
*
*
* @param columnFamilyHandle {@link org.rocksdb.ColumnFamilyHandle}
* instance, or null for the default column family.
* @param property to be fetched. See above for examples
* @return property value
*
* @throws RocksDBException thrown if error happens in underlying
*native library.
*/
public String getProperty(
/* @Nullable */ final ColumnFamilyHandle columnFamilyHandle,
final String property) throws RocksDBException { {code}
Then we can directly export these rt latency number in metrics.
I'd like to introduce 2 rocksdb statistic related configuration.
Then we can customize stats
{code:java}
// code placeholder
Statistics s = new Statistics();
s.setStatsLevel(EXCEPT_TIME_FOR_MUTEX);
currentOptions.setStatsDumpPeriodSec(internalGetOption(RocksDBConfigurableOptions.STATISTIC_DUMP_PERIOD))
.setStatistics(s); {code}
was:
I'd like to introduce 2 rocksdb statistic related configuration.
Then we can customize stats
{code:java}
// code placeholder
Statistics s = new Statistics();
s.setStatsLevel(EXCEPT_TIME_FOR_MUTEX);
currentOptions.setStatsDumpPeriodSec(internalGetOption(RocksDBConfigurableOptions.STATISTIC_DUMP_PERIOD))
.setStatistics(s); {code}
> Exposed Rocksdb statistics in Flink metrics and introduce 2 Rocksdb statistic
> related configuration
> ---
>
> Key: FLINK-33741
> URL: https://issues.apache.org/jira/browse/FLINK-33741
> Project: Flink
> Issue Type: New Feature
>Reporter: xiaogang zhou
>Priority: Major
>
> I think we can also parse the multi-line string of the rocksdb statistics.
> {code:java}
> // code placeholder
> /**
> * DB implements can export properties about their state
> * via this method on a per column family level.
> *
> * If {@code property} is a valid property understood by this DB
> * implementation, fills {@code value} with its current value and
> * returns true. Otherwise returns false.
> *
> * Valid property names include:
> *
> * "rocksdb.num-files-at-levelN" - return the number of files at
> * level N, where N is an ASCII representation of a level
> * number (e.g. "0").
> * "rocksdb.stats" - returns a multi-line string that describes statistics
> * about the internal operation of the DB.
> * "rocksdb.sstables" - returns a multi-line string that describes all
> *of the sstables that make up the db contents.
> *
> *
> * @param columnFamilyHandle {@link org.rocksdb.ColumnFamilyHandle}
> * instance, or null for the default column family.
> * @param property to be fetched. See above for examples
> * @return property value
> *
> * @throws RocksDBException thrown if error happens in underlying
> *native library.
> */
> public String getProperty(
> /* @Nullable */ final ColumnFamilyHandle columnFamilyHandle,
> final String property) throws RocksDBException { {code}
>
> Then we can directly export these rt latency number in metrics.
>
> I'd like to introduce 2 rocksdb statistic related configuration.
> Then we can customize stats
> {code:java}
> // code placeholder
> Statistics s = new Statistics();
> s.setStatsLevel(EXCEPT_TIME_FOR_MUTEX);
> currentOptions.setStatsDumpPeriodSec(internalGetOption(RocksDBConfigurableOptions.STATISTIC_DUMP_PERIOD))
> .setStatistics(s); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33741) Introduce stat dump period and statsLevel configuration
[
https://issues.apache.org/jira/browse/FLINK-33741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xiaogang zhou updated FLINK-33741:
--
Summary: Introduce stat dump period and statsLevel configuration (was:
introduce stat dump period and statsLevel configuration)
> Introduce stat dump period and statsLevel configuration
> ---
>
> Key: FLINK-33741
> URL: https://issues.apache.org/jira/browse/FLINK-33741
> Project: Flink
> Issue Type: New Feature
>Reporter: xiaogang zhou
>Priority: Major
>
> I'd like to introduce 2 rocksdb statistic related configuration.
> Then we can customize stats
> {code:java}
> // code placeholder
> Statistics s = new Statistics();
> s.setStatsLevel(EXCEPT_TIME_FOR_MUTEX);
> currentOptions.setStatsDumpPeriodSec(internalGetOption(RocksDBConfigurableOptions.STATISTIC_DUMP_PERIOD))
> .setStatistics(s); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-25568] Support Elasticsearch Source Connector [flink-connector-elasticsearch]
kaori-seasons commented on PR #62: URL: https://github.com/apache/flink-connector-elasticsearch/pull/62#issuecomment-1848966405 > @kaori-seasons Are you still planning to work on this PR? @MartijnVisser Hello, I just saw your reply recently. Due to work, I did not continue this Pull request. I will start to resolve the conflict this month and request a review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
