[jira] [Updated] (FLINK-31193) The option `table.exec.hive.native-agg-function.enabled` should work at job level when using it in SqlClient side
[ https://issues.apache.org/jira/browse/FLINK-31193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu updated FLINK-31193: -- Description: Sink Flink 1.17, we have implemented the native hive aggregation function, users can enable this optimization by the option `table.exec.hive.native-agg-function.enabled` per-job. The option works well in Table API & HiveServer2Endpoint, but in SQL client, this option can't work per-job, it works at the module level. On the SqlClient side, if we want to use the native hive aggregation function, we need to enable the option first and then load HiveModule, only by doing that we can load the hive native aggregation function. This behavior is inconsistent with the definition of the option itself, and also is inconsistent with the behavior of the Table API and HiveServerEndpoint2, which is a bug. we cannot align the behavior of the option at the moment due to code implementation reasons. If we want to align the behavior, we need to modify the `Module` interface, which is a Public interface. In 1.17 we didn't have time to complete this work, so we need to fix this in 1.18 and make the parameter per-job effective on the SqlClient side as well. was:Sink Flink 1.17, we have implemented the native hive aggregation function, users can enable this optimization by > The option `table.exec.hive.native-agg-function.enabled` should work at job > level when using it in SqlClient side > - > > Key: FLINK-31193 > URL: https://issues.apache.org/jira/browse/FLINK-31193 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Priority: Major > Fix For: 1.18.0 > > > Sink Flink 1.17, we have implemented the native hive aggregation function, > users can enable this optimization by the option > `table.exec.hive.native-agg-function.enabled` per-job. The option works well > in Table API & > HiveServer2Endpoint, but in SQL client, this option can't work per-job, it > works at the module level. > On the SqlClient side, if we want to use the native hive aggregation > function, we need to enable the option first and then load HiveModule, only > by doing that we can load the hive native aggregation function. This behavior > is inconsistent with the definition of the option itself, and also is > inconsistent with the behavior of the Table API and HiveServerEndpoint2, > which is a bug. we cannot align the behavior of the option at the moment due > to code implementation reasons. If we want to align the behavior, we need to > modify the `Module` interface, which is a Public interface. In 1.17 we > didn't have time to complete this work, so we need to fix this in 1.18 and > make the parameter per-job effective on the SqlClient side as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27115: --- Labels: pull-request-available (was: ) > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Improvement >Reporter: zhiyuan >Priority: Major > Labels: pull-request-available > Attachments: image-2023-02-23-11-14-16-479.png, > image-2023-02-23-11-30-39-406.png > > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] huwh opened a new pull request, #22004: [FLINK-27115][blob] Enrich the error log with remote address for blobServer
huwh opened a new pull request, #22004: URL: https://github.com/apache/flink/pull/22004 ## What is the purpose of the change The error log of executing BLOB connection failed does not shows client address. If there are some malicious attacks, it's hard to find the sources. We should log the client address. ## Brief change log - Log the client address when BlobConnection in error. ## Verifying this change - *Manually verified the change by send a `curl` to blob server. The client address will be log.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no ) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31185) Python BroadcastProcessFunction not support side output
[ https://issues.apache.org/jira/browse/FLINK-31185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31185: --- Labels: pull-request-available (was: ) > Python BroadcastProcessFunction not support side output > --- > > Key: FLINK-31185 > URL: https://issues.apache.org/jira/browse/FLINK-31185 > Project: Flink > Issue Type: Bug > Components: API / Python >Affects Versions: 1.16.1 >Reporter: Juntao Hu >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0, 1.16.2 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Vancior opened a new pull request, #22003: [FLINK-31185][Python] Support side-output in broadcast processing
Vancior opened a new pull request, #22003: URL: https://github.com/apache/flink/pull/22003 ## What is the purpose of the change This PR supports using side-output functionality in broadcast processing case in pyflink. ## Brief change log - add operator delegation when processing side-outputs in transformations, since broadcast transformation doesn't have a solid operator - move delegated side-output tags to solid operator in translation stage, making the python operator aware of side-output - adjust embedded-mode code to support side-output in broadcast operator ## Verifying this change This change added tests and can be verified as follows: - `test_co_broadcast_side_output` and `test_keyed_co_broadcast_side_output` in test_data_stream.py ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #212: [FLINK-31125] Flink ML benchmark framework should minimize the source operator overhead
lindong28 commented on code in PR #212:
URL: https://github.com/apache/flink-ml/pull/212#discussion_r1115332664
##
flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/DenseVectorArrayGenerator.java:
##
@@ -42,7 +42,7 @@ protected RowGenerator[] getRowGenerators() {
return new RowGenerator[] {
new RowGenerator(getNumValues(), getSeed()) {
@Override
-protected Row nextRow() {
+protected Row getRow() {
Review Comment:
Yes, I think we need to make this optimization since 2 sec still a big
overhead. And previously I have seen bigger difference (more than 3 sec) after
we make this optimization.
We expect Flink ML's performance to be close to Spark ML. That means we
expect Flink ML's total time for this benchmark to be less than 6 sec after we
finish all optimizations. Then 2 sec is going to take a big percentage of the
total time.
Can you explain why this PR will introduce stability issues? Did you observe
additional stability issues after introducing this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #212: [FLINK-31125] Flink ML benchmark framework should minimize the source operator overhead
lindong28 commented on code in PR #212:
URL: https://github.com/apache/flink-ml/pull/212#discussion_r1115332664
##
flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/DenseVectorArrayGenerator.java:
##
@@ -42,7 +42,7 @@ protected RowGenerator[] getRowGenerators() {
return new RowGenerator[] {
new RowGenerator(getNumValues(), getSeed()) {
@Override
-protected Row nextRow() {
+protected Row getRow() {
Review Comment:
I think we need to make this optimization since 2 sec still a big overhead.
And previously I have seen bigger difference (more than 3 sec) after we make
this optimization.
We expect Flink ML's performance to be close to Spark ML. That means we
expect Flink ML's total time for this benchmark to be less than 6 sec after we
finish all optimizations. Then 2 sec is going to take a big percentage of the
total time.
Can you explain why this PR will introduce stability issues? Did you observe
additional stability issues with this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #212: [FLINK-31125] Flink ML benchmark framework should minimize the source operator overhead
lindong28 commented on code in PR #212:
URL: https://github.com/apache/flink-ml/pull/212#discussion_r1115332664
##
flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/DenseVectorArrayGenerator.java:
##
@@ -42,7 +42,7 @@ protected RowGenerator[] getRowGenerators() {
return new RowGenerator[] {
new RowGenerator(getNumValues(), getSeed()) {
@Override
-protected Row nextRow() {
+protected Row getRow() {
Review Comment:
I think we need to make this optimization since 2 sec still a big overhead.
And previously I have seen bigger difference (more than 3 sec) after we make
this optimization.
We expect Flink ML's performance to be close to Spark ML. That means we
expect Flink ML's total time for this benchmark to be less than 6 sec after we
finish all optimizations. Then 2 sec is going to take a big percentage of the
total time.
Can you explain why this PR will introduce stability issues? Did you observe
additional stability issues after introducing this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31089) pin L0 index in memory can lead to slow memory grow finally lead to memory beyond limit
[ https://issues.apache.org/jira/browse/FLINK-31089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692546#comment-17692546 ] xiaogang zhou commented on FLINK-31089: --- [~yunta] if I turn off the PinTopLevelIndexAndFilter, the task can not run correctly as it takes a lot of time load cache. I also found some rank operator does not has compaction filter in LOG file > pin L0 index in memory can lead to slow memory grow finally lead to memory > beyond limit > --- > > Key: FLINK-31089 > URL: https://issues.apache.org/jira/browse/FLINK-31089 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.16.1 >Reporter: xiaogang zhou >Priority: Major > Attachments: image-2023-02-15-20-26-58-604.png, > image-2023-02-15-20-32-17-993.png, image-2023-02-17-16-48-59-535.png, > l0pin_open.png > > > with the setPinL0FilterAndIndexBlocksInCache true, we can see the pinned > memory kept growing(in the pc blow from 48G-> 50G in about 5 hours). But if > we switch it to false, we can see the pinned memory stay realtive static. In > our environment, a lot of tasks restart due to memory over limit killed by k8s > !image-2023-02-15-20-26-58-604.png|width=899,height=447! > > !image-2023-02-15-20-32-17-993.png|width=853,height=464! > the two graphs are recorded in yesterday and today, which means the data > stream number per second will not differ alot. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31193) The option `table.exec.hive.native-agg-function.enabled` should work at job level when using it in SqlClient side
[ https://issues.apache.org/jira/browse/FLINK-31193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu updated FLINK-31193: -- Description: Sink Flink 1.17, we have implemented the native hive aggregation function, users can enable this optimization by > The option `table.exec.hive.native-agg-function.enabled` should work at job > level when using it in SqlClient side > - > > Key: FLINK-31193 > URL: https://issues.apache.org/jira/browse/FLINK-31193 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.17.0 >Reporter: dalongliu >Priority: Major > Fix For: 1.18.0 > > > Sink Flink 1.17, we have implemented the native hive aggregation function, > users can enable this optimization by -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692543#comment-17692543 ] xzw0223 commented on FLINK-31192: - [~Weijie Guo] thank you > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: xzw0223 >Assignee: xzw0223 >Priority: Major > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on PR #214: URL: https://github.com/apache/flink-ml/pull/214#issuecomment-1441314725 @zhipeng93 Thanks for the review. Can you help take another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692526#comment-17692526 ] Weijie Guo edited comment on FLINK-31192 at 2/23/23 7:31 AM: - [~xzw0223] flink community no longer supports versions lower than 1.15. There is no need to add so many affected versions. BTW, you are assigned. was (Author: weijie guo): [~xzw0223] flink community no longer supports versions lower than 1.15. There is no need to add so many affected versions. > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: xzw0223 >Assignee: xzw0223 >Priority: Major > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo reassigned FLINK-31192: -- Assignee: xzw0223 > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: xzw0223 >Assignee: xzw0223 >Priority: Major > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-18356) flink-table-planner Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692535#comment-17692535
]
Matthias Pohl commented on FLINK-18356:
---
release-1.17:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46435=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=12062
> flink-table-planner Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0
>Reporter: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available, test-stability
> Attachments: 1234.jpg, app-profiling_4.gif,
> image-2023-01-11-22-21-57-784.png, image-2023-01-11-22-22-32-124.png,
> image-2023-02-16-20-18-09-431.png
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31120) ConcurrentModificationException occurred in StringFunctionsITCase.test
[
https://issues.apache.org/jira/browse/FLINK-31120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692534#comment-17692534
]
Matthias Pohl commented on FLINK-31120:
---
This build failure didn't contain the aforementioned fix, yet:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46433=logs=f2c100be-250b-5e85-7bbe-176f68fcddc5=05efd11e-5400-54a4-0d27-a4663be008a9=12384
> ConcurrentModificationException occurred in StringFunctionsITCase.test
> --
>
> Key: FLINK-31120
> URL: https://issues.apache.org/jira/browse/FLINK-31120
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: Shuiqiang Chen
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0, 1.16.2, 1.18.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46255=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12334
> {code}
> Feb 17 04:51:25 [ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 10.725 s <<< FAILURE! - in
> org.apache.flink.table.planner.functions.StringFunctionsITCase
> Feb 17 04:51:25 [ERROR]
> org.apache.flink.table.planner.functions.StringFunctionsITCase.test(TestCase)[4]
> Time elapsed: 4.367 s <<< ERROR!
> Feb 17 04:51:25 org.apache.flink.table.api.TableException: Failed to execute
> sql
> Feb 17 04:51:25 at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeQueryOperation(TableEnvironmentImpl.java:974)
> Feb 17 04:51:25 at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1422)
> Feb 17 04:51:25 at
> org.apache.flink.table.api.internal.TableImpl.execute(TableImpl.java:476)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase$ResultTestItem.test(BuiltInFunctionTestBase.java:354)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase$TestSetSpec.lambda$getTestCase$4(BuiltInFunctionTestBase.java:320)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase$TestCase.execute(BuiltInFunctionTestBase.java:113)
> Feb 17 04:51:25 at
> org.apache.flink.table.planner.functions.BuiltInFunctionTestBase.test(BuiltInFunctionTestBase.java:93)
> Feb 17 04:51:25 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on code in PR #214: URL: https://github.com/apache/flink-ml/pull/214#discussion_r1115310961 ## flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/types/MatrixType.java: ## @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.servable.types; + +import org.apache.flink.annotation.PublicEvolving; + +/** + * A DataType representing a map where all keys have the same BasicType and all values have the same Review Comment: Ah my bad. Thanks for catching this. It is updated to the following doc now. ``` /** A matrix of double values. */ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on code in PR #214:
URL: https://github.com/apache/flink-ml/pull/214#discussion_r1115309864
##
flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/types/MatrixType.java:
##
@@ -0,0 +1,39 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.servable.types;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * A DataType representing a map where all keys have the same BasicType and
all values have the same
+ * BasicType.
+ */
+@PublicEvolving
+public final class MatrixType extends DataType {
Review Comment:
For now, I think it is simpler to keep it as is without distinguishing
between SparseMatrix/DenseMatrix. The code that reads the Matrix can just use
the public API of Matrix without knowing whether it is SparseMatrix or
DenseMatrix. And the code can use `instance of ` if needed.
I expect this to work well since `mleap` uses the same approach and seems to
be OK with it. We can change it if needed in the future.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-31143) Invalid request: offset doesn't match when restarting from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-31143:
--
Issue Type: New Feature (was: Bug)
> Invalid request: offset doesn't match when restarting from a savepoint
> --
>
> Key: FLINK-31143
> URL: https://issues.apache.org/jira/browse/FLINK-31143
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Critical
>
> I tried to run the following case:
> {code:java}
> public static void main(String[] args) throws Exception {
> final String createTableQuery =
> "CREATE TABLE left_table (a int, c varchar) "
> + "WITH ("
> + " 'connector' = 'datagen', "
> + " 'rows-per-second' = '1', "
> + " 'fields.a.kind' = 'sequence', "
> + " 'fields.a.start' = '0', "
> + " 'fields.a.end' = '10'"
> + ");";
> final String selectQuery = "SELECT * FROM left_table;";
> final Configuration initialConfig = new Configuration();
> initialConfig.set(CoreOptions.DEFAULT_PARALLELISM, 1);
> final EnvironmentSettings initialSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(initialConfig)
> .build();
> final TableEnvironment initialTableEnv =
> TableEnvironment.create(initialSettings);
> // create job and consume two results
> initialTableEnv.executeSql(createTableQuery);
> final TableResult tableResult =
> initialTableEnv.sqlQuery(selectQuery).execute();
> tableResult.await();
> System.out.println(tableResultIterator.next());
> System.out.println(tableResultIterator.next());
> // stop job with savepoint
> final String savepointPath;
> try (CloseableIterator tableResultIterator =
> tableResult.collect()) {
> final JobClient jobClient =
>
> tableResult.getJobClient().orElseThrow(IllegalStateException::new);
> final File savepointDirectory = Files.createTempDir();
> savepointPath =
> jobClient
> .stopWithSavepoint(
> true,
> savepointDirectory.getAbsolutePath(),
> SavepointFormatType.CANONICAL)
> .get();
> }
> // restart the very same job from the savepoint
> final SavepointRestoreSettings savepointRestoreSettings =
> SavepointRestoreSettings.forPath(savepointPath, true);
> final Configuration restartConfig = new Configuration(initialConfig);
> SavepointRestoreSettings.toConfiguration(savepointRestoreSettings,
> restartConfig);
> final EnvironmentSettings restartSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(restartConfig)
> .build();
> final TableEnvironment restartTableEnv =
> TableEnvironment.create(restartSettings);
> restartTableEnv.executeSql(createTableQuery);
> restartTableEnv.sqlQuery(selectQuery).execute().print();
> }
> {code}
> h3. Expected behavior
> The job continues omitting the inital two records and starts printing results
> from 2 onwards.
> h3. Observed behavior
> No results are printed. The logs show that an invalid request was handled:
> {code:java}
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction [] -
> Invalid request. Received version = b497a74f-e85c-404b-8df3-1b4b1a0c2de1,
> offset = 0, while expected version = b497a74f-e85c-404b-8df3-1b4b1a0c2de1,
> offset = 1
> {code}
> It looks like the right offset is not picked up from the savepoint (see
> [CollectSinkFunction:411|https://github.com/apache/flink/blob/5ae8cb0503449b07f76d0ab621c3e81734496b26/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/collect/CollectSinkFunction.java#L411]).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31144) Slow scheduling on large-scale batch jobs
[ https://issues.apache.org/jira/browse/FLINK-31144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692530#comment-17692530 ] Weihua Hu commented on FLINK-31144: --- [~zhuzh] That sounds good to me. > Slow scheduling on large-scale batch jobs > -- > > Key: FLINK-31144 > URL: https://issues.apache.org/jira/browse/FLINK-31144 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Julien Tournay >Priority: Major > Attachments: flink-1.17-snapshot-1676473798013.nps, > image-2023-02-21-10-29-49-388.png > > > When executing a complex job graph at high parallelism > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` can > get slow and cause long pauses where the JobManager becomes unresponsive and > all the taskmanagers just wait. I've attached a VisualVM snapshot to > illustrate the problem.[^flink-1.17-snapshot-1676473798013.nps] > At Spotify we have complex jobs where this issue can cause batch "pause" of > 40+ minutes and make the overall execution 30% slower or more. > More importantly this prevent us from running said jobs on larger cluster as > adding resources to the cluster worsen the issue. > We have successfully tested a modified Flink version where > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` was > completely commented and simply returns an empty collection and confirmed it > solves the issue. > In the same spirit as a recent change > ([https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L98-L102)] > there could be a mechanism in place to detect when Flink run into this > specific issue and just skip the call to `getInputLocationFutures` > [https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L105-L108.] > I'm not familiar enough with the internals of Flink to propose a more > advanced fix, however it seems like a configurable threshold on the number of > consumer vertices above which the preferred location is not computed would > do. If this solution is good enough, I'd be happy to submit a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-23810) Print sql when parse failed , which is convenient to find error sql from multiple executed sql
[ https://issues.apache.org/jira/browse/FLINK-23810 ]
hehuiyuan deleted comment on FLINK-23810:
---
was (Author: hehuiyuan):
[~jark] ,[~twalthr] , I can try to do it if necessary.
> Print sql when parse failed , which is convenient to find error sql from
> multiple executed sql
> --
>
> Key: FLINK-23810
> URL: https://issues.apache.org/jira/browse/FLINK-23810
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: hehuiyuan
>Priority: Major
> Labels: pull-request-available
>
> Print sql when parse failed , which is convenient to find error sql.
>
> {code:java}
> public SqlNode parse(String sql) {
> try {
> SqlParser parser = SqlParser.create(sql, config);
> return parser.parseStmt();
> } catch (SqlParseException e) {
> throw new SqlParserException("SQL parse failed. " + e.getMessage(),
> e);
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on code in PR #214:
URL: https://github.com/apache/flink-ml/pull/214#discussion_r1115308290
##
flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/api/DataFrame.java:
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.servable.api;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.servable.types.DataType;
+
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A DataFrame consists of some number of rows, each of which has the same
list of column names and
+ * data types.
+ *
+ * All values in a column must have the same data type: integer, float,
string etc.
+ */
+@PublicEvolving
+public class DataFrame {
+
+private final List columnNames;
+private final List dataTypes;
+private final List rows;
+
+/**
+ * The given columnNames and dataTypes should be mutable in order for
TransformerServable
+ * classes to update DataFrame with the serving results.
+ */
+public DataFrame(List columnNames, List dataTypes,
List rows) {
+int numColumns = columnNames.size();
+if (dataTypes.size() != numColumns) {
+throw new IllegalArgumentException(
+String.format(
+"The number of data types %d is different from the
number of column names %d.",
+dataTypes.size(), numColumns));
+}
+for (Row row : rows) {
+if (row.size() != numColumns) {
+throw new IllegalArgumentException(
+String.format(
+"The row size %d is different from the number
of column names %d.",
+row.size(), numColumns));
+}
+}
+
+this.columnNames = columnNames;
+this.dataTypes = dataTypes;
+this.rows = rows;
+}
+
+/** Returns a list of the names of all the columns in this DataFrame. */
+public List getColumnNames() {
+return columnNames;
+}
+
+/**
+ * Returns the index of the column with the given name.
+ *
+ * @throws IllegalArgumentException if the column is not present in this
table
+ */
+public int getIndex(String name) {
+int index = columnNames.indexOf(name);
+if (index == -1) {
+throw new IllegalArgumentException("Failed to find the column with
the given name.");
+}
+return index;
+}
+
+/**
+ * Returns the data type of the column with the given name.
+ *
+ * @throws IllegalArgumentException if the column is not present in this
table
+ */
+public DataType getDataType(String name) {
+int index = getIndex(name);
+return dataTypes.get(index);
+}
+
+/**
+ * Adds to this DataFrame a column with the given name, data type, and
values.
+ *
+ * @throws IllegalArgumentException if the number of values is different
from the number of
+ * rows.
+ */
+public DataFrame addColumn(String columnName, DataType dataType,
List values) {
+if (values.size() != rows.size()) {
+throw new RuntimeException(
+String.format(
+"The number of values %d is different from the
number of rows %d.",
+values.size(), rows.size()));
+}
+columnNames.add(columnName);
+dataTypes.add(dataType);
+
+int rowSize = -1;
+Iterator iter = values.iterator();
+for (Row row : rows) {
+if (rowSize < 0) {
Review Comment:
Thanks for catching this! It is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-23810) Print sql when parse failed , which is convenient to find error sql from multiple executed sql
[
https://issues.apache.org/jira/browse/FLINK-23810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692531#comment-17692531
]
hehuiyuan commented on FLINK-23810:
---
Hi [~jark] [~twalthr] , whether to add the detail sql?
> Print sql when parse failed , which is convenient to find error sql from
> multiple executed sql
> --
>
> Key: FLINK-23810
> URL: https://issues.apache.org/jira/browse/FLINK-23810
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Reporter: hehuiyuan
>Priority: Major
> Labels: pull-request-available
>
> Print sql when parse failed , which is convenient to find error sql.
>
> {code:java}
> public SqlNode parse(String sql) {
> try {
> SqlParser parser = SqlParser.create(sql, config);
> return parser.parseStmt();
> } catch (SqlParseException e) {
> throw new SqlParserException("SQL parse failed. " + e.getMessage(),
> e);
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on code in PR #214:
URL: https://github.com/apache/flink-ml/pull/214#discussion_r1115308117
##
flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/api/TransformerServable.java:
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.servable.api;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.param.WithParams;
+
+/**
+ * A TransformerServable takes a DataFrame as input and produces a DataFrame
as the result. It can
+ * be used to encode online inference computation logic.
+ *
+ * NOTE: Every TransformerServable subclass should have a no-arg
constructor.
+ *
+ * NOTE: Every TransformerServable subclass should implement a static
method with signature
+ * {@code static T load(String path) throws IOException;}, where {@code T}
refers to the concrete
+ * subclass. This static method should instantiate a new TransformerServable
instance based on the
+ * data read from the given path.
+ *
+ * @param The class type of the TransformerServable implementation itself.
+ */
+@PublicEvolving
+public interface TransformerServable> extends
WithParams {
+/**
+ * Applies the TransformerServable on the given input DataFrame and
returns the result
+ * DataFrame.
+ *
+ * @param input the input data
Review Comment:
Do you mean to change it to `@param input The input data`?
I am not sure whether it is the right stye to use. The examples provided in
[1] [2] as well as the existing doc for e.g. `AlgoOperator#transform` uses the
lower case character. And the usage is also inconsistent in Flink codebase. For
example, `DataStream#transform` uses lower case character.
How about we use the lower case for now and have a followup PR to change the
existing Java doc later?
[1]
https://www.oracle.com/technical-resources/articles/java/javadoc-tool.html
[2] https://www.baeldung.com/javadoc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on a diff in pull request #21999: [FLINK-29816][streaming] Fix the bug that SteamTask skip handle exception during restoring.
RocMarshal commented on code in PR #21999:
URL: https://github.com/apache/flink/pull/21999#discussion_r1115303685
##
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskITCase.java:
##
@@ -110,6 +118,47 @@ public void testFailInEndOfConstructor() throws Exception {
.build(EXECUTOR_RESOURCE.getExecutor()));
}
+@Test
+public void
streamTaskAsyncExceptionHandlerHandleExceptionInInitializingDuration()
+throws Exception {
+MockEnvironment mockEnvironment = MockEnvironment.builder().build();
+RuntimeException expectedException = new RuntimeException("RUNTIME
EXCEPTION");
+
+
mockEnvironment.setExpectedExternalFailureCause(AsynchronousException.class);
+final String expectedErrorMessage = "EXPECTED_ERROR MESSAGE";
+
+NoOpStreamTask noOpStreamTask =
+new NoOpStreamTask(mockEnvironment) {
+volatile boolean handledAsyncException = false;
+
+@Override
+protected void init() throws Exception {
+super.init();
+
+CompletableFuture.runAsync(
+() -> {
+this.handleAsyncException(
+expectedErrorMessage,
expectedException);
+handledAsyncException = true;
+});
+while (!handledAsyncException) {
+ThreadUtils.sleep(Duration.ofMillis(10));
+}
+}
+};
+noOpStreamTask.invoke();
+
+Optional actualExternalFailureCause =
+mockEnvironment.getActualExternalFailureCause();
+final Throwable actualException =
+actualExternalFailureCause.orElseThrow(
+() -> new AssertionError("Expected exceptional
completion"));
+
+assertThat(actualException, instanceOf(AsynchronousException.class));
+assertThat(actualException.getMessage(), is("EXPECTED_ERROR MESSAGE"));
Review Comment:
```suggestion
assertThat(actualException.getMessage(), is(expectedErrorMessage));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on code in PR #214:
URL: https://github.com/apache/flink-ml/pull/214#discussion_r1115304116
##
flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/api/DataFrame.java:
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.servable.api;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.servable.types.DataType;
+
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A DataFrame consists of some number of rows, each of which has the same
list of column names and
+ * data types.
+ *
+ * All values in a column must have the same data type: integer, float,
string etc.
+ */
+@PublicEvolving
+public class DataFrame {
+
+private final List columnNames;
+private final List dataTypes;
+private final List rows;
+
+/**
+ * The given columnNames and dataTypes should be mutable in order for
TransformerServable
+ * classes to update DataFrame with the serving results.
+ */
+public DataFrame(List columnNames, List dataTypes,
List rows) {
+int numColumns = columnNames.size();
+if (dataTypes.size() != numColumns) {
+throw new IllegalArgumentException(
+String.format(
+"The number of data types %d is different from the
number of column names %d.",
+dataTypes.size(), numColumns));
+}
+for (Row row : rows) {
+if (row.size() != numColumns) {
+throw new IllegalArgumentException(
+String.format(
+"The row size %d is different from the number
of column names %d.",
+row.size(), numColumns));
+}
+}
+
+this.columnNames = columnNames;
+this.dataTypes = dataTypes;
+this.rows = rows;
+}
+
+/** Returns a list of the names of all the columns in this DataFrame. */
+public List getColumnNames() {
+return columnNames;
+}
+
+/**
+ * Returns the index of the column with the given name.
+ *
+ * @throws IllegalArgumentException if the column is not present in this
table
+ */
+public int getIndex(String name) {
+int index = columnNames.indexOf(name);
+if (index == -1) {
+throw new IllegalArgumentException("Failed to find the column with
the given name.");
Review Comment:
Thanks. It is fixed now.
##
flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/api/ModelServable.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.servable.api;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+import java.io.IOException;
+import java.io.InputStream;
+
+/**
+ * A ModelServable is a TransformerServable with the extra API to set model
data.
+ *
+ * @param The class type of the ModelServable implementation itself.
+ */
+@PublicEvolving
+public interface ModelServable> extends
TransformerServable {
+
+/** Sets model data using the serialized model data from the given input
streams. */
+default T setModelData(InputStream... modelDataInputs) throws IOException {
+throw new UnsupportedOperationException("this operation is not
supported");
Review Comment:
Thanks. It is fixed now.
--
This is an
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #214: [FLINK-31127] Add public API classes for FLIP-289
lindong28 commented on code in PR #214: URL: https://github.com/apache/flink-ml/pull/214#discussion_r1115303911 ## flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/api/DataFrame.java: ## @@ -0,0 +1,132 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.servable.api; + +import org.apache.flink.annotation.PublicEvolving; +import org.apache.flink.ml.servable.types.DataType; + +import java.util.Iterator; +import java.util.List; + +/** + * A DataFrame consists of some number of rows, each of which has the same list of column names and Review Comment: Thanks. It is fixed now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #212: [FLINK-31125] Flink ML benchmark framework should minimize the source operator overhead
zhipeng93 commented on code in PR #212:
URL: https://github.com/apache/flink-ml/pull/212#discussion_r1115302692
##
flink-ml-benchmark/src/main/java/org/apache/flink/ml/benchmark/datagenerator/common/DenseVectorArrayGenerator.java:
##
@@ -42,7 +42,7 @@ protected RowGenerator[] getRowGenerators() {
return new RowGenerator[] {
new RowGenerator(getNumValues(), getSeed()) {
@Override
-protected Row nextRow() {
+protected Row getRow() {
Review Comment:
Thanks for the update. As generating 10^8 rows takes ~2s, do you think
generating data is still a big overhead that needs to be addressed? Since
reusing some of the rows does introduce some stability issues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31144) Slow scheduling on large-scale batch jobs
[ https://issues.apache.org/jira/browse/FLINK-31144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692527#comment-17692527 ] Zhu Zhu commented on FLINK-31144: - I suspect the slowness is caused by the N^2 complexity to compute the input locations when there are N upstream task and N downstream tasks. If so, as long as N is not too large, e.g. not larger than MAX_DISTINCT_CONSUMERS_TO_CONSIDER=8, the cost of input location computation should be acceptable. Also, if there are too many distinct consumers, input locality would make none sense. As [~huwh] mentioned, it's hard for users to decides a proper threshold for each job. It's also inconvenient if users had to decide whether to enable input locality or not. Therefore, I prefer to let Flink decide it automatically for users, like the proposed change above. > Slow scheduling on large-scale batch jobs > -- > > Key: FLINK-31144 > URL: https://issues.apache.org/jira/browse/FLINK-31144 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Julien Tournay >Priority: Major > Attachments: flink-1.17-snapshot-1676473798013.nps, > image-2023-02-21-10-29-49-388.png > > > When executing a complex job graph at high parallelism > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` can > get slow and cause long pauses where the JobManager becomes unresponsive and > all the taskmanagers just wait. I've attached a VisualVM snapshot to > illustrate the problem.[^flink-1.17-snapshot-1676473798013.nps] > At Spotify we have complex jobs where this issue can cause batch "pause" of > 40+ minutes and make the overall execution 30% slower or more. > More importantly this prevent us from running said jobs on larger cluster as > adding resources to the cluster worsen the issue. > We have successfully tested a modified Flink version where > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` was > completely commented and simply returns an empty collection and confirmed it > solves the issue. > In the same spirit as a recent change > ([https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L98-L102)] > there could be a mechanism in place to detect when Flink run into this > specific issue and just skip the call to `getInputLocationFutures` > [https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L105-L108.] > I'm not familiar enough with the internals of Flink to propose a more > advanced fix, however it seems like a configurable threshold on the number of > consumer vertices above which the preferred location is not computed would > do. If this solution is good enough, I'd be happy to submit a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31144) Slow scheduling on large-scale batch jobs
[ https://issues.apache.org/jira/browse/FLINK-31144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692527#comment-17692527 ] Zhu Zhu edited comment on FLINK-31144 at 2/23/23 7:06 AM: -- I suspect the slowness is caused by the N^2 complexity to compute the input locations when there are N upstream task and N downstream tasks. If so, as long as N is not too large, e.g. not larger than MAX_DISTINCT_CONSUMERS_TO_CONSIDER=8, the cost of input location computation should be acceptable. Also, if there are too many distinct consumers, input locality would make none sense. As [~huwh] mentioned, it's hard for users to decides a proper threshold for each job. It's also inconvenient if users had to decide whether to enable input locality or not. Therefore, I prefer to let Flink decide it automatically for users, like the proposed change above. was (Author: zhuzh): I suspect the slowness is caused by the N^2 complexity to compute the input locations when there are N upstream task and N downstream tasks. If so, as long as N is not too large, e.g. not larger than MAX_DISTINCT_CONSUMERS_TO_CONSIDER=8, the cost of input location computation should be acceptable. Also, if there are too many distinct consumers, input locality would make none sense. As [~huwh] mentioned, it's hard for users to decides a proper threshold for each job. It's also inconvenient if users had to decide whether to enable input locality or not. Therefore, I prefer to let Flink decide it automatically for users, like the proposed change above. > Slow scheduling on large-scale batch jobs > -- > > Key: FLINK-31144 > URL: https://issues.apache.org/jira/browse/FLINK-31144 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Reporter: Julien Tournay >Priority: Major > Attachments: flink-1.17-snapshot-1676473798013.nps, > image-2023-02-21-10-29-49-388.png > > > When executing a complex job graph at high parallelism > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` can > get slow and cause long pauses where the JobManager becomes unresponsive and > all the taskmanagers just wait. I've attached a VisualVM snapshot to > illustrate the problem.[^flink-1.17-snapshot-1676473798013.nps] > At Spotify we have complex jobs where this issue can cause batch "pause" of > 40+ minutes and make the overall execution 30% slower or more. > More importantly this prevent us from running said jobs on larger cluster as > adding resources to the cluster worsen the issue. > We have successfully tested a modified Flink version where > `DefaultPreferredLocationsRetriever.getPreferredLocationsBasedOnInputs` was > completely commented and simply returns an empty collection and confirmed it > solves the issue. > In the same spirit as a recent change > ([https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L98-L102)] > there could be a mechanism in place to detect when Flink run into this > specific issue and just skip the call to `getInputLocationFutures` > [https://github.com/apache/flink/blob/43f419d0eccba86ecc8040fa6f521148f1e358ff/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultPreferredLocationsRetriever.java#L105-L108.] > I'm not familiar enough with the internals of Flink to propose a more > advanced fix, however it seems like a configurable threshold on the number of > consumer vertices above which the preferred location is not computed would > do. If this solution is good enough, I'd be happy to submit a PR. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692526#comment-17692526 ] Weijie Guo commented on FLINK-31192: [~xzw0223] flink community no longer supports versions lower than 1.15. There is no need to add so many affected versions. > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: xzw0223 >Priority: Major > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31194) Introduces savepoint mechanism of Table Store
[ https://issues.apache.org/jira/browse/FLINK-31194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Jiang updated FLINK-31194: --- Summary: Introduces savepoint mechanism of Table Store (was: ntroduces savepoint mechanism of Table Store) > Introduces savepoint mechanism of Table Store > - > > Key: FLINK-31194 > URL: https://issues.apache.org/jira/browse/FLINK-31194 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Nicholas Jiang >Priority: Major > Fix For: table-store-0.4.0 > > > Disaster Recovery is very much mission critical for any software. Especially > when it comes to data systems, the impact could be very serious leading to > delay in business decisions or even wrong business decisions at times. Flink > Table Store could introduce savepoint mechanism to assist users in recovering > data from a previous state. > As the name suggest, "savepoint" saves the table as of the snapshot, so that > it lets you restore the table to this savepoint at a later point in snapshot > if need be. Care is taken to ensure cleaner will not clean up any files that > are savepointed. On similar lines, savepoint cannot be triggered on a > snapshot that is already cleaned up. In simpler terms, this is synonymous to > taking a backup, just that we don't make a new copy of the table, but just > save the state of the table elegantly so that we can restore it later when in > need. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-31192: --- Fix Version/s: (was: 1.11.0) (was: 1.12.0) (was: 1.13.0) (was: 1.14.0) (was: 1.15.0) (was: 1.16.0) (was: 1.16.1) > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: xzw0223 >Priority: Major > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] tsreaper merged pull request #551: [FLINK-31179] Make data structures serializable
tsreaper merged PR #551: URL: https://github.com/apache/flink-table-store/pull/551 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-31192: --- Affects Version/s: 1.15.3 1.17.0 (was: 1.16.0) > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: xzw0223 >Priority: Major > Fix For: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.16.1 > > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31194) ntroduces savepoint mechanism of Table Store
Nicholas Jiang created FLINK-31194: -- Summary: ntroduces savepoint mechanism of Table Store Key: FLINK-31194 URL: https://issues.apache.org/jira/browse/FLINK-31194 Project: Flink Issue Type: New Feature Components: Table Store Affects Versions: table-store-0.4.0 Reporter: Nicholas Jiang Fix For: table-store-0.4.0 Disaster Recovery is very much mission critical for any software. Especially when it comes to data systems, the impact could be very serious leading to delay in business decisions or even wrong business decisions at times. Flink Table Store could introduce savepoint mechanism to assist users in recovering data from a previous state. As the name suggest, "savepoint" saves the table as of the snapshot, so that it lets you restore the table to this savepoint at a later point in snapshot if need be. Care is taken to ensure cleaner will not clean up any files that are savepointed. On similar lines, savepoint cannot be triggered on a snapshot that is already cleaned up. In simpler terms, this is synonymous to taking a backup, just that we don't make a new copy of the table, but just save the state of the table elegantly so that we can restore it later when in need. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #22002: [FLINK-30989][runtime] Some config options related to sorting and spilling are not valid.
flinkbot commented on PR #22002: URL: https://github.com/apache/flink/pull/22002#issuecomment-1441286850 ## CI report: * b54aa98baf5ccd1c3558d2066e1170c9880f8afe UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa opened a new pull request, #22002: [FLINK-30989][runtime] Some config options related to sorting and spilling are not valid.
reswqa opened a new pull request, #22002: URL: https://github.com/apache/flink/pull/22002 ## What is the purpose of the change *For some config options like `taskmanager.runtime.sort-spilling-threshold` ,`taskmanager.runtime.max-fan` ... etc. used for sorting and spilling, we get this option value from [JobConfiguration](https://github.com/apache/flink/blob/1c0870ae08730688706540b999c04b2f4c4498ee/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/sort/SortingDataInput.java#L118). In fact, these parameters should be only available in the `TaskManagerConfiguration` now.* ## Brief change log - *Using `TaskManagerConfiguration` to load these config options.* ## Verifying this change Manual testing. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31193) The option `table.exec.hive.native-agg-function.enabled` should work at job level when using it in SqlClient side
dalongliu created FLINK-31193: - Summary: The option `table.exec.hive.native-agg-function.enabled` should work at job level when using it in SqlClient side Key: FLINK-31193 URL: https://issues.apache.org/jira/browse/FLINK-31193 Project: Flink Issue Type: Sub-task Components: Connectors / Hive Affects Versions: 1.17.0 Reporter: dalongliu Fix For: 1.18.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Aitozi commented on pull request #20775: [FLINK-29088][table] Extract the common project to enhance source reuse
Aitozi commented on PR #20775: URL: https://github.com/apache/flink/pull/20775#issuecomment-1441277370 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #214: [FLINK-31127] Add public API classes for FLIP-289
zhipeng93 commented on code in PR #214:
URL: https://github.com/apache/flink-ml/pull/214#discussion_r1115277590
##
flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/api/DataFrame.java:
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.servable.api;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.ml.servable.types.DataType;
+
+import java.util.Iterator;
+import java.util.List;
+
+/**
+ * A DataFrame consists of some number of rows, each of which has the same
list of column names and
+ * data types.
+ *
+ * All values in a column must have the same data type: integer, float,
string etc.
+ */
+@PublicEvolving
+public class DataFrame {
+
+private final List columnNames;
+private final List dataTypes;
+private final List rows;
+
+/**
+ * The given columnNames and dataTypes should be mutable in order for
TransformerServable
+ * classes to update DataFrame with the serving results.
+ */
+public DataFrame(List columnNames, List dataTypes,
List rows) {
+int numColumns = columnNames.size();
+if (dataTypes.size() != numColumns) {
+throw new IllegalArgumentException(
+String.format(
+"The number of data types %d is different from the
number of column names %d.",
+dataTypes.size(), numColumns));
+}
+for (Row row : rows) {
+if (row.size() != numColumns) {
+throw new IllegalArgumentException(
+String.format(
+"The row size %d is different from the number
of column names %d.",
+row.size(), numColumns));
+}
+}
+
+this.columnNames = columnNames;
+this.dataTypes = dataTypes;
+this.rows = rows;
+}
+
+/** Returns a list of the names of all the columns in this DataFrame. */
+public List getColumnNames() {
+return columnNames;
+}
+
+/**
+ * Returns the index of the column with the given name.
+ *
+ * @throws IllegalArgumentException if the column is not present in this
table
+ */
+public int getIndex(String name) {
+int index = columnNames.indexOf(name);
+if (index == -1) {
+throw new IllegalArgumentException("Failed to find the column with
the given name.");
+}
+return index;
+}
+
+/**
+ * Returns the data type of the column with the given name.
+ *
+ * @throws IllegalArgumentException if the column is not present in this
table
+ */
+public DataType getDataType(String name) {
+int index = getIndex(name);
+return dataTypes.get(index);
+}
+
+/**
+ * Adds to this DataFrame a column with the given name, data type, and
values.
+ *
+ * @throws IllegalArgumentException if the number of values is different
from the number of
+ * rows.
+ */
+public DataFrame addColumn(String columnName, DataType dataType,
List values) {
+if (values.size() != rows.size()) {
+throw new RuntimeException(
+String.format(
+"The number of values %d is different from the
number of rows %d.",
+values.size(), rows.size()));
+}
+columnNames.add(columnName);
+dataTypes.add(dataType);
+
+int rowSize = -1;
+Iterator iter = values.iterator();
+for (Row row : rows) {
+if (rowSize < 0) {
Review Comment:
The check for row size here seems uncessary since we already checked them in
the constructor.
##
flink-ml-servable-core/src/main/java/org/apache/flink/ml/servable/api/DataFrame.java:
##
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may
[jira] [Commented] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692518#comment-17692518 ] xzw0223 commented on FLINK-31192: - Add one thing : It is added that during snapshot restoration and snapshot, it is still necessary to traverse all data in Deque for saving and restoration > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.16.0, 1.16.1 >Reporter: xzw0223 >Priority: Major > Fix For: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.16.1 > > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-28086) Table Store Catalog supports partition methods
[ https://issues.apache.org/jira/browse/FLINK-28086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-28086: Assignee: (was: Nicholas Jiang) > Table Store Catalog supports partition methods > -- > > Key: FLINK-28086 > URL: https://issues.apache.org/jira/browse/FLINK-28086 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Minor > Fix For: table-store-0.4.0 > > > Table Store Catalog can support: > * listPartitions > * listPartitionsByFilter > * getPartition > * partitionExists > * dropPartition -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] reswqa closed pull request #21797: [FLINK-29845] Using System.nanoTime instead of currentTimeMillis in T…
reswqa closed pull request #21797: [FLINK-29845] Using System.nanoTime instead of currentTimeMillis in T… URL: https://github.com/apache/flink/pull/21797 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa closed pull request #21953: [FLINK-30989][runtime] Some config options related to sorting and spilling are not valid.
reswqa closed pull request #21953: [FLINK-30989][runtime] Some config options related to sorting and spilling are not valid. URL: https://github.com/apache/flink/pull/21953 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa closed pull request #20229: [FLINK-28399] Refactor CheckedThread to accept runnable via constructor
reswqa closed pull request #20229: [FLINK-28399] Refactor CheckedThread to accept runnable via constructor URL: https://github.com/apache/flink/pull/20229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa closed pull request #20024: [FLINK-28125] Promote leadership changing logs to INFO level
reswqa closed pull request #20024: [FLINK-28125] Promote leadership changing logs to INFO level URL: https://github.com/apache/flink/pull/20024 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JunRuiLee commented on pull request #21943: [FLINK-31077][runtime] Mark pending checkpoint onCompletionPromise complete only after the completed checkpoint is added to the store.
JunRuiLee commented on PR #21943: URL: https://github.com/apache/flink/pull/21943#issuecomment-1441270605 @gaoyunhaii Thanks for CR and I've addressed comment, PTAL! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xzw0223 updated FLINK-31192: Fix Version/s: 1.15.0 1.14.0 1.13.0 1.12.0 1.11.0 > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.16.0, 1.16.1 >Reporter: xzw0223 >Priority: Major > Fix For: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.16.1 > > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692514#comment-17692514 ] xzw0223 commented on FLINK-31192: - [~Weijie Guo] All versions, it seems, have been around since birth. > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.16.0, 1.16.1 >Reporter: xzw0223 >Priority: Major > Fix For: 1.16.0, 1.16.1 > > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692515#comment-17692515 ] xzw0223 commented on FLINK-31192: - I'd like to solve this problem. May I have a ticket [~Weijie Guo] > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.16.0, 1.16.1 >Reporter: xzw0223 >Priority: Major > Fix For: 1.16.0, 1.16.1 > > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692510#comment-17692510 ] Weijie Guo commented on FLINK-31192: [~xzw0223] Thanks for reporting this, does this problem only exist in 1.16? If not, you'd better adjust the affected version. > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.16.0, 1.16.1 >Reporter: xzw0223 >Priority: Major > Fix For: 1.16.0, 1.16.1 > > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] gaoyunhaii commented on a diff in pull request #21943: [FLINK-31077][runtime] Mark pending checkpoint onCompletionPromise complete only after the completed checkpoint is added to the
gaoyunhaii commented on code in PR #21943:
URL: https://github.com/apache/flink/pull/21943#discussion_r1115275981
##
flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java:
##
@@ -1314,13 +1314,12 @@ private void
completePendingCheckpoint(PendingCheckpoint pendingCheckpoint)
if (!props.isSavepoint()) {
lastSubsumed =
addCompletedCheckpointToStoreAndSubsumeOldest(
-checkpointId,
-completedCheckpoint,
-
pendingCheckpoint.getCheckpointPlan().getTasksToCommitTo());
+checkpointId, completedCheckpoint,
pendingCheckpoint);
} else {
lastSubsumed = null;
}
+
pendingCheckpoint.getCompletionFuture().complete(completedCheckpoint);
Review Comment:
I have a small concern here current `CompleteExceptionally` is not in the
same level with the complete. In this case, there is risk that if we change
logic here, there might be case that the future is not done finally.
To not change the order of "complete future" -> "report the checkpoint", we
might also try marking the checkpoint in the catch block in this method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31192) dataGen takes too long to initialize under sequence
[ https://issues.apache.org/jira/browse/FLINK-31192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692508#comment-17692508 ] xzw0223 commented on FLINK-31192: - [~mapohl] I created a new issue to discuss the problem of slow datagen initialization. We can discuss how this problem can be better solved. > dataGen takes too long to initialize under sequence > --- > > Key: FLINK-31192 > URL: https://issues.apache.org/jira/browse/FLINK-31192 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.16.0, 1.16.1 >Reporter: xzw0223 >Priority: Major > Fix For: 1.16.0, 1.16.1 > > > The SequenceGenerator preloads all sequence values in open. If the > totalElement number is too large, it will take too long. > [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] > The reason is that the capacity of the Deque will be expanded twice when the > current capacity is full, and the array copy is required, which is > time-consuming. > > Here's what I think : > do not preload the full amount of data on Sequence, and generate a piece of > data each time next is called to solve the problem of slow initialization > caused by loading full amount of data. > record the currently sent Sequence position through the checkpoint, and > continue to send data through the recorded position after an abnormal restart > to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31190) Supports Spark call procedure command on Table Store
[ https://issues.apache.org/jira/browse/FLINK-31190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-31190: Assignee: Nicholas Jiang > Supports Spark call procedure command on Table Store > > > Key: FLINK-31190 > URL: https://issues.apache.org/jira/browse/FLINK-31190 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Nicholas Jiang >Assignee: Nicholas Jiang >Priority: Major > Fix For: table-store-0.4.0 > > > At present Hudi and Iceberg supports the Spark call procedure command to > execute the table service action etc. Flink Table Store could also support > Spark call procedure command to run compaction etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692507#comment-17692507 ] Xintong Song commented on FLINK-27115: -- Log the client address only in error cases sounds good to me. > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Improvement >Reporter: zhiyuan >Priority: Major > Attachments: image-2023-02-23-11-14-16-479.png, > image-2023-02-23-11-30-39-406.png > > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31191) VectorIndexer should check whether doublesByColumn is null before snapshot
[ https://issues.apache.org/jira/browse/FLINK-31191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31191: --- Labels: pull-request-available (was: ) > VectorIndexer should check whether doublesByColumn is null before snapshot > -- > > Key: FLINK-31191 > URL: https://issues.apache.org/jira/browse/FLINK-31191 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Affects Versions: ml-2.2.0 >Reporter: Zhipeng Zhang >Priority: Major > Labels: pull-request-available > > Currently VectorIndexer would lead to NPE when doing checkpoint. It should > check whether `doublesByColumn` is null before calling snapshot. > > logview: > [https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039] > details: > > > [735|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:736]Caused > by: java.lang.NullPointerException > [736|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:737] > at > org.apache.flink.ml.feature.vectorindexer.VectorIndexer$ComputeDistinctDoublesOperator.convertToListArray(VectorIndexer.java:232) > > [737|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:738] > at > org.apache.flink.ml.feature.vectorindexer.VectorIndexer$ComputeDistinctDoublesOperator.snapshotState(VectorIndexer.java:228) > > [738|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:739] > at > org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:222) > > [739|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:740] > ... 33 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] zhipeng93 opened a new pull request, #217: [FLINK-31191] Fix null pointer exception in VectorIndexer
zhipeng93 opened a new pull request, #217: URL: https://github.com/apache/flink-ml/pull/217 ## What is the purpose of the change - Fix NPE in VectorIndexer. ## Brief change log - Add check for `doublesByColumn` when doing snapshot. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-27115: - Affects Version/s: (was: 1.12.7) > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Improvement >Reporter: zhiyuan >Priority: Major > Attachments: image-2023-02-23-11-14-16-479.png, > image-2023-02-23-11-30-39-406.png > > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-27115: - Issue Type: Improvement (was: Bug) > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Improvement >Affects Versions: 1.12.7 >Reporter: zhiyuan >Priority: Major > Attachments: image-2023-02-23-11-14-16-479.png, > image-2023-02-23-11-30-39-406.png > > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31192) dataGen takes too long to initialize under sequence
xzw0223 created FLINK-31192: --- Summary: dataGen takes too long to initialize under sequence Key: FLINK-31192 URL: https://issues.apache.org/jira/browse/FLINK-31192 Project: Flink Issue Type: Improvement Affects Versions: 1.16.1, 1.16.0 Reporter: xzw0223 Fix For: 1.16.1, 1.16.0 The SequenceGenerator preloads all sequence values in open. If the totalElement number is too large, it will take too long. [https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/datagen/SequenceGenerator.java#L91] The reason is that the capacity of the Deque will be expanded twice when the current capacity is full, and the array copy is required, which is time-consuming. Here's what I think : do not preload the full amount of data on Sequence, and generate a piece of data each time next is called to solve the problem of slow initialization caused by loading full amount of data. record the currently sent Sequence position through the checkpoint, and continue to send data through the recorded position after an abnormal restart to ensure fault tolerance -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31182) CompiledPlan cannot deserialize BridgingSqlFunction with MissingTypeStrategy
[
https://issues.apache.org/jira/browse/FLINK-31182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692450#comment-17692450
]
Jane Chan edited comment on FLINK-31182 at 2/23/23 6:08 AM:
[~Sergey Nuyanzin] Yes, the check on the output type inference strategy was
introduced in FunctionCatalogTableOperator in FLINK-15487 for FLIP-65 a long
time ago.
I guess it worked well before FLIP-190 was introduced
because`LogicalUnnestRule` will infer the output type and convert UNNEST to
EXPLODE. (see [LogicalUnnestRule
L#99|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalUnnestRule.scala#L99]).
As a result, although the BuiltinFunctionDefinition for UNNEST always has a
"MISSING" type inference, it does not affect the execution.
However, after CompiledPlan is introduced, RexNodeJsonDeserializer relies on
the OperatorTable to lookup functions, which always get a static placeholder
(i.e. MISSING) for UNNEST, and due to this check, the deserialization failed.
Could you help to take a look, cc [~godfreyhe] and [~twalthr]
was (Author: qingyue):
[~Sergey Nuyanzin] Yes, the check on the output type inference strategy was
introduced in FunctionCatalogTableOperator in FLINK-15487 for FLIP-65 a long
time ago.
I guess it worked well before FLIP-190 was introduced
because`LogicalUnnestRule` will infer the output type and convert UNNEST to
EXPLODE. (see [LogicalUnnestRule
L#99|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalUnnestRule.scala#L99]).
As a result, although the BuiltinFunctionDefinition for UNNEST always has a
"MISSING" type inference, it does not affect the execution.
However, after CompiledPlan is introduced, RexNodeJsonDeserializer relies on
the OperatorTable to lookup functions, which always get a static placeholder
(i.e. MISSING) for UNNEST, and due to this check, the deserialization failed.
> CompiledPlan cannot deserialize BridgingSqlFunction with MissingTypeStrategy
>
>
> Key: FLINK-31182
> URL: https://issues.apache.org/jira/browse/FLINK-31182
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.17.0, 1.18.0, 1.17.1
>Reporter: Jane Chan
>Priority: Major
> Attachments: screenshot-1.png
>
>
> This issue is reported from the [user mail
> list|https://lists.apache.org/thread/y6fgzyx330omhkr40376knw8k4oczz3s].
> The stacktrace is
> {code:java}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlCaused by: org.apache.flink.table.api.TableException: Could not resolve
> internal system function '$UNNEST_ROWS$1'. This is a bug, please file an
> issue.
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserializeInternalFunction(RexNodeJsonDeserializer.java:392)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserializeSqlOperator(RexNodeJsonDeserializer.java:337)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserializeCall(RexNodeJsonDeserializer.java:307)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserialize(RexNodeJsonDeserializer.java:146)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserialize(RexNodeJsonDeserializer.java:128)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserialize(RexNodeJsonDeserializer.java:115)
> {code}
> The root cause is that although ModuleManager can resolve '$UNNEST_ROWS$1',
> the output type strategy is "Missing"; as a result,
> FunctionCatalogOperatorTable#convertToBridgingSqlFunction returns empty.
> !screenshot-1.png|width=675,height=295!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31191) VectorIndexer should check whether doublesByColumn is null before snapshot
Zhipeng Zhang created FLINK-31191: - Summary: VectorIndexer should check whether doublesByColumn is null before snapshot Key: FLINK-31191 URL: https://issues.apache.org/jira/browse/FLINK-31191 Project: Flink Issue Type: Bug Components: Library / Machine Learning Affects Versions: ml-2.2.0 Reporter: Zhipeng Zhang Currently VectorIndexer would lead to NPE when doing checkpoint. It should check whether `doublesByColumn` is null before calling snapshot. logview: [https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039] details: [735|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:736]Caused by: java.lang.NullPointerException [736|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:737] at org.apache.flink.ml.feature.vectorindexer.VectorIndexer$ComputeDistinctDoublesOperator.convertToListArray(VectorIndexer.java:232) [737|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:738] at org.apache.flink.ml.feature.vectorindexer.VectorIndexer$ComputeDistinctDoublesOperator.snapshotState(VectorIndexer.java:228) [738|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:739] at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:222) [739|https://github.com/apache/flink-ml/actions/runs/4249415318/jobs/7389547039#step:4:740] ... 33 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31124) Add it case for HiveTableSink speculative execution
[ https://issues.apache.org/jira/browse/FLINK-31124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu closed FLINK-31124. --- Fix Version/s: 1.17.0 Resolution: Done master: db6e01e028ec69d7702d3371d6b3db253aafbbf8 release-1.17: c94a0799016ae7f8d6e348930124936bc71a61aa > Add it case for HiveTableSink speculative execution > --- > > Key: FLINK-31124 > URL: https://issues.apache.org/jira/browse/FLINK-31124 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination, Tests >Reporter: Biao Liu >Assignee: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > The part of HiveTableSink has supported speculative execution in > https://issues.apache.org/jira/browse/FLINK-30823. We would like to add some > integration test cases for this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zhuzhurk closed pull request #21962: [FLINK-31124][Connectors/Hive] Add IT case for HiveTableSink speculative execution
zhuzhurk closed pull request #21962: [FLINK-31124][Connectors/Hive] Add IT case for HiveTableSink speculative execution URL: https://github.com/apache/flink/pull/21962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] FangYongs commented on pull request #21979: [FLINK-30810] Rework CliClientITCase extends AbstractStatementTestBase
FangYongs commented on PR #21979: URL: https://github.com/apache/flink/pull/21979#issuecomment-1441244618 Hi @fsk119 Please help to review this PR when you're free, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on a diff in pull request #21999: [FLINK-29816][streaming] Fix the bug that SteamTask skip handle exception during restoring.
RocMarshal commented on code in PR #21999:
URL: https://github.com/apache/flink/pull/21999#discussion_r1115242031
##
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskITCase.java:
##
@@ -110,6 +118,42 @@ public void testFailInEndOfConstructor() throws Exception {
.build(EXECUTOR_RESOURCE.getExecutor()));
}
+@Test
+public void
streamTaskAsyncExceptionHandlerHandleExceptionInInitializingDuration()
+throws Exception {
+MockEnvironment mockEnvironment = MockEnvironment.builder().build();
+RuntimeException expectedException = new RuntimeException("RUNTIME
EXCEPTION");
+
+
mockEnvironment.setExpectedExternalFailureCause(AsynchronousException.class);
+final String expectedErrorMessage = "EXPECTED_ERROR MESSAGE";
+
+NoOpStreamTask noOpStreamTask =
+new NoOpStreamTask(mockEnvironment) {
+@Override
+protected void init() throws Exception {
+super.init();
+CompletableFuture.runAsync(
+() -> {
+assertFalse(this.isRunning());
+assertTrue(this.isInitializing());
+this.handleAsyncException(
+expectedErrorMessage,
expectedException);
+});
+}
Review Comment:
Should we make the main thread of NoOpStreamTask sleep a small duration
or add a flag to ensure that The `runAsync` operation must run in the span
(isRestoring = true, isRunning=false) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] zhuangchong opened a new pull request, #553: Unify the version of `kafka` in pom.
zhuangchong opened a new pull request, #553: URL: https://github.com/apache/flink-table-store/pull/553 Unify the version of `kafka` in pom. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on a diff in pull request #21999: [FLINK-29816][streaming] Fix the bug that SteamTask skip handle exception during restoring.
RocMarshal commented on code in PR #21999:
URL: https://github.com/apache/flink/pull/21999#discussion_r1115242031
##
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskITCase.java:
##
@@ -110,6 +118,42 @@ public void testFailInEndOfConstructor() throws Exception {
.build(EXECUTOR_RESOURCE.getExecutor()));
}
+@Test
+public void
streamTaskAsyncExceptionHandlerHandleExceptionInInitializingDuration()
+throws Exception {
+MockEnvironment mockEnvironment = MockEnvironment.builder().build();
+RuntimeException expectedException = new RuntimeException("RUNTIME
EXCEPTION");
+
+
mockEnvironment.setExpectedExternalFailureCause(AsynchronousException.class);
+final String expectedErrorMessage = "EXPECTED_ERROR MESSAGE";
+
+NoOpStreamTask noOpStreamTask =
+new NoOpStreamTask(mockEnvironment) {
+@Override
+protected void init() throws Exception {
+super.init();
+CompletableFuture.runAsync(
+() -> {
+assertFalse(this.isRunning());
+assertTrue(this.isInitializing());
+this.handleAsyncException(
+expectedErrorMessage,
expectedException);
+});
+}
Review Comment:
Should the make thread of NoOpStreamTask sleep a small duration
Or add a flag to ensure that The `runAsync` operation must run in the span
(isRestoring = true, isRunning=false) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31190) Supports Spark call procedure command on Table Store
[ https://issues.apache.org/jira/browse/FLINK-31190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692494#comment-17692494 ] Nicholas Jiang commented on FLINK-31190: [~lzljs3620320], could you assign this ticket to me? I would like to support Spark call procedure command which refers to the implementation of Hudi and Iceberg. > Supports Spark call procedure command on Table Store > > > Key: FLINK-31190 > URL: https://issues.apache.org/jira/browse/FLINK-31190 > Project: Flink > Issue Type: New Feature > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Nicholas Jiang >Priority: Major > Fix For: table-store-0.4.0 > > > At present Hudi and Iceberg supports the Spark call procedure command to > execute the table service action etc. Flink Table Store could also support > Spark call procedure command to run compaction etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31190) Supports Spark call procedure command on Table Store
Nicholas Jiang created FLINK-31190: -- Summary: Supports Spark call procedure command on Table Store Key: FLINK-31190 URL: https://issues.apache.org/jira/browse/FLINK-31190 Project: Flink Issue Type: New Feature Components: Table Store Affects Versions: table-store-0.4.0 Reporter: Nicholas Jiang Fix For: table-store-0.4.0 At present Hudi and Iceberg supports the Spark call procedure command to execute the table service action etc. Flink Table Store could also support Spark call procedure command to run compaction etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31173) Fix several bugs in flink-ml-iteration module
[ https://issues.apache.org/jira/browse/FLINK-31173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhipeng Zhang updated FLINK-31173: -- Description: In flink-ml-iteration, there are several bugs as follows: # TailOperator should have one input operator. We have added a Tail operator to increment the epoch watermark at each iteration. We have made an assumption that each Tail operator have only one input and did not align the epoch watermarks from different inputs. This assumption might not be true if the input is an `union`. # ReplayOperator should replay the records when it receives max epochwatermark. Currently ReplayOperator does not replay the records when it receives the max epochwatermark. However, it is inconsistent with the HeadOperator. As HeadOperator will always forward the record to downstreams. # ProxyOperatorStateBackend does not correctly initialize the state descriptor. was: In flink-ml-iteration, there are several bugs as follows: # TailOperator should have one input operator. We have added a Tail operator to increment the epoch watermark at each iteration. We have made an assumption that each Tail operator have only one input and did not align the epoch watermarks from different inputs. This assumption might not be true if the input is an `union`. # ReplayOperator should replay the records when it receives max epochwatermark. Currently ReplayOperator does not replay the records when it receives the max epochwatermark. However, it is inconsistent with the HeadOperator. As HeadOperator will always forward the record to downstreams. # ProxyOperatorStateBackend does not correctly initialize the state descriptor. > Fix several bugs in flink-ml-iteration module > - > > Key: FLINK-31173 > URL: https://issues.apache.org/jira/browse/FLINK-31173 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Affects Versions: ml-2.0.0, ml-2.1.0, ml-2.2.0 >Reporter: Zhipeng Zhang >Priority: Major > > In flink-ml-iteration, there are several bugs as follows: > # TailOperator should have one input operator. We have added a Tail operator > to increment the epoch watermark at each iteration. We have made an > assumption that each Tail operator have only one input and did not align the > epoch watermarks from different inputs. This assumption might not be true if > the input is an `union`. > # ReplayOperator should replay the records when it receives max > epochwatermark. Currently ReplayOperator does not replay the records when it > receives the max epochwatermark. However, it is inconsistent with the > HeadOperator. As HeadOperator will always forward the record to downstreams. > # ProxyOperatorStateBackend does not correctly initialize the state > descriptor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31173) Fix several bugs in flink-ml-iteration module
[ https://issues.apache.org/jira/browse/FLINK-31173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhipeng Zhang reassigned FLINK-31173: - Assignee: Zhipeng Zhang > Fix several bugs in flink-ml-iteration module > - > > Key: FLINK-31173 > URL: https://issues.apache.org/jira/browse/FLINK-31173 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Affects Versions: ml-2.0.0, ml-2.1.0, ml-2.2.0 >Reporter: Zhipeng Zhang >Assignee: Zhipeng Zhang >Priority: Major > > In flink-ml-iteration, there are several bugs as follows: > # TailOperator should have one input operator. We have added a Tail operator > to increment the epoch watermark at each iteration. We have made an > assumption that each Tail operator have only one input and did not align the > epoch watermarks from different inputs. This assumption might not be true if > the input is an `union`. > # ReplayOperator should replay the records when it receives max > epochwatermark. Currently ReplayOperator does not replay the records when it > receives the max epochwatermark. However, it is inconsistent with the > HeadOperator. As HeadOperator will always forward the record to downstreams. > # ProxyOperatorStateBackend does not correctly initialize the state > descriptor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31173) Fix several bugs in flink-ml-iteration module
[ https://issues.apache.org/jira/browse/FLINK-31173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhipeng Zhang updated FLINK-31173: -- Description: In flink-ml-iteration, there are several bugs as follows: # TailOperator should have one input operator. We have added a Tail operator to increment the epoch watermark at each iteration. We have made an assumption that each Tail operator have only one input and did not align the epoch watermarks from different inputs. This assumption might not be true if the input is an `union`. # ReplayOperator should replay the records when it receives max epochwatermark. Currently ReplayOperator does not replay the records when it receives the max epochwatermark. However, it is inconsistent with the HeadOperator. As HeadOperator will always forward the record to downstreams. # ProxyOperatorStateBackend does not correctly initialize the state descriptor. was: In flink-ml-iteration, we have added a Tail operator to increment the epoch watermark at each iteration. We have made an assumption that each Tail operator have only one input and did not align the epoch watermarks from different inputs. This assumption might not be true if the input is an `union`. I propose to add an explicit check to prevent from having multiple input for the TailOperator. If a the input does contain multiple inputs, then users can add a map operator after union. Summary: Fix several bugs in flink-ml-iteration module (was: TailOperator should only have one input) > Fix several bugs in flink-ml-iteration module > - > > Key: FLINK-31173 > URL: https://issues.apache.org/jira/browse/FLINK-31173 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Affects Versions: ml-2.0.0, ml-2.1.0, ml-2.2.0 >Reporter: Zhipeng Zhang >Priority: Major > > In flink-ml-iteration, there are several bugs as follows: > > # TailOperator should have one input operator. We have added a Tail operator > to increment the epoch watermark at each iteration. We have made an > assumption that each Tail operator have only one input and did not align the > epoch watermarks from different inputs. This assumption might not be true if > the input is an `union`. > # ReplayOperator should replay the records when it receives max > epochwatermark. Currently ReplayOperator does not replay the records when it > receives the max epochwatermark. However, it is inconsistent with the > HeadOperator. As HeadOperator will always forward the record to downstreams. > # ProxyOperatorStateBackend does not correctly initialize the state > descriptor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-31176) correct the description of sql gateway configuration
[ https://issues.apache.org/jira/browse/FLINK-31176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang reassigned FLINK-31176: - Assignee: wangkang > correct the description of sql gateway configuration > > > Key: FLINK-31176 > URL: https://issues.apache.org/jira/browse/FLINK-31176 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Gateway >Affects Versions: 1.16.0 >Reporter: wangkang >Assignee: wangkang >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > Attachments: image-2023-02-22-11-17-08-611.png > > > correct the description of sql gateway configuration: > 1.sql-gateway.session.idle-timeout 、sql-gateway.session.check-interval > description in SqlGatewayServiceConfigOptions > 2.GetSessionConfigHeaders and TriggerSessionHeartbeatHeaders class description > !image-2023-02-22-11-17-08-611.png|width=717,height=289! > when setting sql-gateway.session.idle-timeout to negative value,SqlGateway > will throw NumberFormatException,beacause the TimeUtils.pasDuration method > doesn't support the negative value,so we should remove the 'or negative > value' description -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-31176) correct the description of sql gateway configuration
[ https://issues.apache.org/jira/browse/FLINK-31176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang closed FLINK-31176. - Fix Version/s: 1.18.0 Resolution: Fixed Merged into master: b252abe179b5b4458e5306f7bdf8e484ac61d515 > correct the description of sql gateway configuration > > > Key: FLINK-31176 > URL: https://issues.apache.org/jira/browse/FLINK-31176 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Gateway >Affects Versions: 1.16.0 >Reporter: wangkang >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > Attachments: image-2023-02-22-11-17-08-611.png > > > correct the description of sql gateway configuration: > 1.sql-gateway.session.idle-timeout 、sql-gateway.session.check-interval > description in SqlGatewayServiceConfigOptions > 2.GetSessionConfigHeaders and TriggerSessionHeartbeatHeaders class description > !image-2023-02-22-11-17-08-611.png|width=717,height=289! > when setting sql-gateway.session.idle-timeout to negative value,SqlGateway > will throw NumberFormatException,beacause the TimeUtils.pasDuration method > doesn't support the negative value,so we should remove the 'or negative > value' description -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] fsk119 merged pull request #21983: [FLINK-31176][sql-gateway] correct the description of sql gateway configuration
fsk119 merged PR #21983: URL: https://github.com/apache/flink/pull/21983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Rruop commented on pull request #21968: [FLINK-31123][connector/files] Add IT cases for FileSink speculative …
Rruop commented on PR #21968: URL: https://github.com/apache/flink/pull/21968#issuecomment-1441191875 > LGTM. Thanks for addressing the comments! @Rruop Would you squash the commits? Thanks for your patiently reviewing, I will squash all the commits into one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31189) Allow special handle of less frequent values in StringIndexer
[ https://issues.apache.org/jira/browse/FLINK-31189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fan Hong updated FLINK-31189: - Description: Real-world datasets often contain categorical features with millions of distinct values, some of which may only appear a few times. To maximize the performance of certain algorithms, it is important to treat these less frequent values properly. A popular approach is to put them to a special index, as is done in sklearn's OneHotEncoder [1]. [1] https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html was: In real-world datasets, categorical features may have millions of distinct values, while some of them may only occur few times. Special handling of less frequent values can bring performance increase in some algorithms. One > Allow special handle of less frequent values in StringIndexer > - > > Key: FLINK-31189 > URL: https://issues.apache.org/jira/browse/FLINK-31189 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Fan Hong >Priority: Major > > Real-world datasets often contain categorical features with millions of > distinct values, some of which may only appear a few times. To maximize the > performance of certain algorithms, it is important to treat these less > frequent values properly. A popular approach is to put them to a special > index, as is done in sklearn's OneHotEncoder [1]. > > [1] > https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31189) Allow special handle of less frequent values in StringIndexer
[ https://issues.apache.org/jira/browse/FLINK-31189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fan Hong updated FLINK-31189: - Summary: Allow special handle of less frequent values in StringIndexer (was: Allow ignore less frequent values in StringIndexer) > Allow special handle of less frequent values in StringIndexer > - > > Key: FLINK-31189 > URL: https://issues.apache.org/jira/browse/FLINK-31189 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Fan Hong >Priority: Major > > In real-world datasets, categorical features may have millions of distinct > values, while some of them may only occur few times. Special handling of less > frequent values can bring performance increase in some algorithms. > > One -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31189) Allow ignore less frequent values in StringIndexer
[ https://issues.apache.org/jira/browse/FLINK-31189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fan Hong updated FLINK-31189: - Description: In real-world datasets, categorical features may have millions of distinct values, while some of them may only occur few times. Special handling of less frequent values can bring performance increase in some algorithms. One > Allow ignore less frequent values in StringIndexer > -- > > Key: FLINK-31189 > URL: https://issues.apache.org/jira/browse/FLINK-31189 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Fan Hong >Priority: Major > > In real-world datasets, categorical features may have millions of distinct > values, while some of them may only occur few times. Special handling of less > frequent values can bring performance increase in some algorithms. > > One -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Aitozi commented on pull request #20775: [FLINK-29088][table] Extract the common project to enhance source reuse
Aitozi commented on PR #20775: URL: https://github.com/apache/flink/pull/20775#issuecomment-1441176867 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692473#comment-17692473 ] Weihua Hu commented on FLINK-27115: --- I tried to send a request to the netty server and it will log the source address. !image-2023-02-23-11-30-39-406.png! > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Bug >Affects Versions: 1.12.7 >Reporter: zhiyuan >Priority: Major > Attachments: image-2023-02-23-11-14-16-479.png, > image-2023-02-23-11-30-39-406.png > > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31189) Allow ignore less frequent values in StringIndexer
Fan Hong created FLINK-31189: Summary: Allow ignore less frequent values in StringIndexer Key: FLINK-31189 URL: https://issues.apache.org/jira/browse/FLINK-31189 Project: Flink Issue Type: Improvement Components: Library / Machine Learning Reporter: Fan Hong -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu updated FLINK-27115: -- Attachment: image-2023-02-23-11-30-39-406.png > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Bug >Affects Versions: 1.12.7 >Reporter: zhiyuan >Priority: Major > Attachments: image-2023-02-23-11-14-16-479.png, > image-2023-02-23-11-30-39-406.png > > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[
https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692471#comment-17692471
]
Weihua Hu commented on FLINK-27115:
---
Sorry, the screenshot has been attached.
{quote}I cannot see the screenshot. Is this exception causing problems like job
failure or it's just a warning / error in logs?
{quote}
It's just a ERROR log, will not trigger job failure.
{quote}That sounds like a malicious attack.
{quote}
For our scenario they are internal security scan requests. It took some time to
confirm this. It would be helpful if it could log who is sending these
requests.
{quote}Do we need this log in other common scenarios where such undesired
requests do not exist?
{quote}
No, we only need enrich the error log with source address.
> Error while executing BLOB connection. java.io.IOException: Unknown operation
> 80
>
>
> Key: FLINK-27115
> URL: https://issues.apache.org/jira/browse/FLINK-27115
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.12.7
>Reporter: zhiyuan
>Priority: Major
> Attachments: image-2023-02-23-11-14-16-479.png
>
>
> hi,
> I have a Flink SQL job running online. Every morning, I will report the
> following screenshot error. I have enabled the debug log to check and find
> nothing
> version 1.12
> !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true!
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] gj199575 commented on pull request #20467: [FLINK-28825][k8s] Add K8S pod scheduler into Kubernetes options
gj199575 commented on PR #20467: URL: https://github.com/apache/flink/pull/20467#issuecomment-1441172829 I will handle this PR as @bzhaoopenstack successor 。 And there are two PR(https://github.com/apache/flink/pull/20498 https://github.com/apache/flink/pull/20636) is replenish of this PR。 And I also commit a PR as replenish 。 https://github.com/bzhaoopenstack/flink/pull/5 I wish more friends can give some advices。 Thanks. I wish we can merge these PR， which can help flink work on k8s better。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-31136) SQL Client Gateway mode should not read read execution config
[ https://issues.apache.org/jira/browse/FLINK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengkai Fang closed FLINK-31136. - Resolution: Fixed > SQL Client Gateway mode should not read read execution config > - > > Key: FLINK-31136 > URL: https://issues.apache.org/jira/browse/FLINK-31136 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.17.0 >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31136) SQL Client Gateway mode should not read read execution config
[ https://issues.apache.org/jira/browse/FLINK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691889#comment-17691889 ] Shengkai Fang edited comment on FLINK-31136 at 2/23/23 3:24 AM: Merged into release-1.17: db15e1e2014f02cbeb58d8a4fee1befdbd5a3ac8 Merged into master: dec3ba078decbdc212a6ea16ad8728aa7409d9c3 was (Author: fsk119): Mreged into release-1.17: db15e1e2014f02cbeb58d8a4fee1befdbd5a3ac8 > SQL Client Gateway mode should not read read execution config > - > > Key: FLINK-31136 > URL: https://issues.apache.org/jira/browse/FLINK-31136 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.17.0 >Reporter: Shengkai Fang >Assignee: Shengkai Fang >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] fsk119 merged pull request #21987: Cherry pick FLINK-31136 to master
fsk119 merged PR #21987: URL: https://github.com/apache/flink/pull/21987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31143) Invalid request: offset doesn't match when restarting from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692463#comment-17692463
]
Caizhi Weng edited comment on FLINK-31143 at 2/23/23 3:21 AM:
--
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
It is actually not easy to implement this. The biggest problem is that collect
client is a stateful client and it stores some records not consumed by the user
yet. If we want to support {{collect}} + savepoint, we will first need to find
out where to store these unconsumed records. That is, we might need to
introduce something like client state backend.
A less user-friendly but more feasible solution is that we require the user to
consume all records in the {{collect}} iterator after a savepoint, before
closing the iterator and starting a new job. If the user does not obey this
requirement some records might be lost.
In any aspect, supporting {{collect}} + savepoint will be a new feature. As we
are very close to releasing I think it is proper to introduce it (if needed) in
future versions.
was (Author: tsreaper):
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
It is actually not easy to implement this. The biggest problem is that collect
client is a stateful client and it stores some records not consumed by the user
yet. If we want to support {{collect}} + savepoint, we will first need to find
out where to store these unconsumed records. That is, we might need to
introduce something like client state backend.
> Invalid request: offset doesn't match when restarting from a savepoint
> --
>
> Key: FLINK-31143
> URL: https://issues.apache.org/jira/browse/FLINK-31143
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Critical
>
> I tried to run the following case:
> {code:java}
> public static void main(String[] args) throws Exception {
> final String createTableQuery =
> "CREATE TABLE left_table (a int, c varchar) "
> + "WITH ("
> + " 'connector' = 'datagen', "
> + " 'rows-per-second' = '1', "
> + " 'fields.a.kind' = 'sequence', "
> + " 'fields.a.start' = '0', "
> + " 'fields.a.end' = '10'"
> + ");";
> final String selectQuery = "SELECT * FROM left_table;";
> final Configuration initialConfig = new Configuration();
> initialConfig.set(CoreOptions.DEFAULT_PARALLELISM, 1);
> final EnvironmentSettings initialSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(initialConfig)
> .build();
> final TableEnvironment initialTableEnv =
> TableEnvironment.create(initialSettings);
> // create job and consume two results
> initialTableEnv.executeSql(createTableQuery);
> final TableResult tableResult =
> initialTableEnv.sqlQuery(selectQuery).execute();
> tableResult.await();
> System.out.println(tableResultIterator.next());
> System.out.println(tableResultIterator.next());
> // stop job with savepoint
> final String savepointPath;
> try (CloseableIterator tableResultIterator =
> tableResult.collect()) {
> final JobClient jobClient =
>
> tableResult.getJobClient().orElseThrow(IllegalStateException::new);
> final File savepointDirectory = Files.createTempDir();
> savepointPath =
> jobClient
> .stopWithSavepoint(
> true,
> savepointDirectory.getAbsolutePath(),
> SavepointFormatType.CANONICAL)
> .get();
> }
> // restart the very same job from the savepoint
> final SavepointRestoreSettings savepointRestoreSettings =
> SavepointRestoreSettings.forPath(savepointPath, true);
> final Configuration restartConfig = new Configuration(initialConfig);
> SavepointRestoreSettings.toConfiguration(savepointRestoreSettings,
> restartConfig);
> final EnvironmentSettings restartSettings =
>
[jira] [Comment Edited] (FLINK-31143) Invalid request: offset doesn't match when restarting from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692463#comment-17692463
]
Caizhi Weng edited comment on FLINK-31143 at 2/23/23 3:17 AM:
--
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
It is actually not easy to implement this. The biggest problem is that collect
client is a stateful client and it stores some records not consumed by the user
yet. If we want to support {{collect}} + savepoint, we will first need to find
out where to store these unconsumed records. That is, we might need to
introduce something like client state backend.
was (Author: tsreaper):
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
It is actually not east to implement this. The biggest problem is that collect
client is a stateful client and it stores some records not consumed by the user
yet. If we want to support {{collect}} + savepoint, we will first need to find
out where to store these unconsumed records. That is, we might need to
introduce something like client state backend.
> Invalid request: offset doesn't match when restarting from a savepoint
> --
>
> Key: FLINK-31143
> URL: https://issues.apache.org/jira/browse/FLINK-31143
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Critical
>
> I tried to run the following case:
> {code:java}
> public static void main(String[] args) throws Exception {
> final String createTableQuery =
> "CREATE TABLE left_table (a int, c varchar) "
> + "WITH ("
> + " 'connector' = 'datagen', "
> + " 'rows-per-second' = '1', "
> + " 'fields.a.kind' = 'sequence', "
> + " 'fields.a.start' = '0', "
> + " 'fields.a.end' = '10'"
> + ");";
> final String selectQuery = "SELECT * FROM left_table;";
> final Configuration initialConfig = new Configuration();
> initialConfig.set(CoreOptions.DEFAULT_PARALLELISM, 1);
> final EnvironmentSettings initialSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(initialConfig)
> .build();
> final TableEnvironment initialTableEnv =
> TableEnvironment.create(initialSettings);
> // create job and consume two results
> initialTableEnv.executeSql(createTableQuery);
> final TableResult tableResult =
> initialTableEnv.sqlQuery(selectQuery).execute();
> tableResult.await();
> System.out.println(tableResultIterator.next());
> System.out.println(tableResultIterator.next());
> // stop job with savepoint
> final String savepointPath;
> try (CloseableIterator tableResultIterator =
> tableResult.collect()) {
> final JobClient jobClient =
>
> tableResult.getJobClient().orElseThrow(IllegalStateException::new);
> final File savepointDirectory = Files.createTempDir();
> savepointPath =
> jobClient
> .stopWithSavepoint(
> true,
> savepointDirectory.getAbsolutePath(),
> SavepointFormatType.CANONICAL)
> .get();
> }
> // restart the very same job from the savepoint
> final SavepointRestoreSettings savepointRestoreSettings =
> SavepointRestoreSettings.forPath(savepointPath, true);
> final Configuration restartConfig = new Configuration(initialConfig);
> SavepointRestoreSettings.toConfiguration(savepointRestoreSettings,
> restartConfig);
> final EnvironmentSettings restartSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(restartConfig)
> .build();
> final TableEnvironment restartTableEnv =
> TableEnvironment.create(restartSettings);
> restartTableEnv.executeSql(createTableQuery);
> restartTableEnv.sqlQuery(selectQuery).execute().print();
> }
> {code}
> h3. Expected behavior
> The job continues omitting the
[jira] [Comment Edited] (FLINK-31143) Invalid request: offset doesn't match when restarting from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692463#comment-17692463
]
Caizhi Weng edited comment on FLINK-31143 at 2/23/23 3:17 AM:
--
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
It is actually not east to implement this. The biggest problem is that collect
client is a stateful client and it stores some records not consumed by the user
yet. If we want to support {{collect}} + savepoint, we will first need to find
out where to store these unconsumed records. That is, we might need to
introduce something like client state backend.
was (Author: tsreaper):
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
> Invalid request: offset doesn't match when restarting from a savepoint
> --
>
> Key: FLINK-31143
> URL: https://issues.apache.org/jira/browse/FLINK-31143
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Critical
>
> I tried to run the following case:
> {code:java}
> public static void main(String[] args) throws Exception {
> final String createTableQuery =
> "CREATE TABLE left_table (a int, c varchar) "
> + "WITH ("
> + " 'connector' = 'datagen', "
> + " 'rows-per-second' = '1', "
> + " 'fields.a.kind' = 'sequence', "
> + " 'fields.a.start' = '0', "
> + " 'fields.a.end' = '10'"
> + ");";
> final String selectQuery = "SELECT * FROM left_table;";
> final Configuration initialConfig = new Configuration();
> initialConfig.set(CoreOptions.DEFAULT_PARALLELISM, 1);
> final EnvironmentSettings initialSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(initialConfig)
> .build();
> final TableEnvironment initialTableEnv =
> TableEnvironment.create(initialSettings);
> // create job and consume two results
> initialTableEnv.executeSql(createTableQuery);
> final TableResult tableResult =
> initialTableEnv.sqlQuery(selectQuery).execute();
> tableResult.await();
> System.out.println(tableResultIterator.next());
> System.out.println(tableResultIterator.next());
> // stop job with savepoint
> final String savepointPath;
> try (CloseableIterator tableResultIterator =
> tableResult.collect()) {
> final JobClient jobClient =
>
> tableResult.getJobClient().orElseThrow(IllegalStateException::new);
> final File savepointDirectory = Files.createTempDir();
> savepointPath =
> jobClient
> .stopWithSavepoint(
> true,
> savepointDirectory.getAbsolutePath(),
> SavepointFormatType.CANONICAL)
> .get();
> }
> // restart the very same job from the savepoint
> final SavepointRestoreSettings savepointRestoreSettings =
> SavepointRestoreSettings.forPath(savepointPath, true);
> final Configuration restartConfig = new Configuration(initialConfig);
> SavepointRestoreSettings.toConfiguration(savepointRestoreSettings,
> restartConfig);
> final EnvironmentSettings restartSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(restartConfig)
> .build();
> final TableEnvironment restartTableEnv =
> TableEnvironment.create(restartSettings);
> restartTableEnv.executeSql(createTableQuery);
> restartTableEnv.sqlQuery(selectQuery).execute().print();
> }
> {code}
> h3. Expected behavior
> The job continues omitting the inital two records and starts printing results
> from 2 onwards.
> h3. Observed behavior
> No results are printed. The logs show that an invalid request was handled:
> {code:java}
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction [] -
> Invalid request. Received version = b497a74f-e85c-404b-8df3-1b4b1a0c2de1,
> offset = 0, while expected
[jira] [Updated] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu updated FLINK-27115: -- Attachment: image-2023-02-23-11-14-16-479.png > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Bug >Affects Versions: 1.12.7 >Reporter: zhiyuan >Priority: Major > Attachments: image-2023-02-23-11-14-16-479.png > > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31143) Invalid request: offset doesn't match when restarting from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692463#comment-17692463
]
Caizhi Weng edited comment on FLINK-31143 at 2/23/23 3:14 AM:
--
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
was (Author: tsreaper):
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not
supported. I can implement this but I think this is actually a new feature
rather than a bug fix. As we're very close to releasing I think it might be
more proper to introduce this feature in 1.17.1. What do you think?
> Invalid request: offset doesn't match when restarting from a savepoint
> --
>
> Key: FLINK-31143
> URL: https://issues.apache.org/jira/browse/FLINK-31143
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Critical
>
> I tried to run the following case:
> {code:java}
> public static void main(String[] args) throws Exception {
> final String createTableQuery =
> "CREATE TABLE left_table (a int, c varchar) "
> + "WITH ("
> + " 'connector' = 'datagen', "
> + " 'rows-per-second' = '1', "
> + " 'fields.a.kind' = 'sequence', "
> + " 'fields.a.start' = '0', "
> + " 'fields.a.end' = '10'"
> + ");";
> final String selectQuery = "SELECT * FROM left_table;";
> final Configuration initialConfig = new Configuration();
> initialConfig.set(CoreOptions.DEFAULT_PARALLELISM, 1);
> final EnvironmentSettings initialSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(initialConfig)
> .build();
> final TableEnvironment initialTableEnv =
> TableEnvironment.create(initialSettings);
> // create job and consume two results
> initialTableEnv.executeSql(createTableQuery);
> final TableResult tableResult =
> initialTableEnv.sqlQuery(selectQuery).execute();
> tableResult.await();
> System.out.println(tableResultIterator.next());
> System.out.println(tableResultIterator.next());
> // stop job with savepoint
> final String savepointPath;
> try (CloseableIterator tableResultIterator =
> tableResult.collect()) {
> final JobClient jobClient =
>
> tableResult.getJobClient().orElseThrow(IllegalStateException::new);
> final File savepointDirectory = Files.createTempDir();
> savepointPath =
> jobClient
> .stopWithSavepoint(
> true,
> savepointDirectory.getAbsolutePath(),
> SavepointFormatType.CANONICAL)
> .get();
> }
> // restart the very same job from the savepoint
> final SavepointRestoreSettings savepointRestoreSettings =
> SavepointRestoreSettings.forPath(savepointPath, true);
> final Configuration restartConfig = new Configuration(initialConfig);
> SavepointRestoreSettings.toConfiguration(savepointRestoreSettings,
> restartConfig);
> final EnvironmentSettings restartSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(restartConfig)
> .build();
> final TableEnvironment restartTableEnv =
> TableEnvironment.create(restartSettings);
> restartTableEnv.executeSql(createTableQuery);
> restartTableEnv.sqlQuery(selectQuery).execute().print();
> }
> {code}
> h3. Expected behavior
> The job continues omitting the inital two records and starts printing results
> from 2 onwards.
> h3. Observed behavior
> No results are printed. The logs show that an invalid request was handled:
> {code:java}
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction [] -
> Invalid request. Received version = b497a74f-e85c-404b-8df3-1b4b1a0c2de1,
> offset = 0, while expected version = b497a74f-e85c-404b-8df3-1b4b1a0c2de1,
> offset = 1
> {code}
> It looks like the right offset is not picked up from the savepoint (see
>
[jira] [Comment Edited] (FLINK-31143) Invalid request: offset doesn't match when restarting from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692463#comment-17692463
]
Caizhi Weng edited comment on FLINK-31143 at 2/23/23 3:13 AM:
--
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not
supported. I can implement this but I think this is actually a new feature
rather than a bug fix. As we're very close to releasing I think it might be
more proper to introduce this feature in 1.17.1. What do you think?
was (Author: tsreaper):
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
However it isn't hard to implement because the collect sink has sent everything
needed to the client, just that client didn't handle the case. I can implement
this but I think this is actually a new feature rather than a bug fix. As we're
very close to releasing I think it might be more proper to introduce this
feature in 1.17.1. What do you think?
> Invalid request: offset doesn't match when restarting from a savepoint
> --
>
> Key: FLINK-31143
> URL: https://issues.apache.org/jira/browse/FLINK-31143
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Critical
>
> I tried to run the following case:
> {code:java}
> public static void main(String[] args) throws Exception {
> final String createTableQuery =
> "CREATE TABLE left_table (a int, c varchar) "
> + "WITH ("
> + " 'connector' = 'datagen', "
> + " 'rows-per-second' = '1', "
> + " 'fields.a.kind' = 'sequence', "
> + " 'fields.a.start' = '0', "
> + " 'fields.a.end' = '10'"
> + ");";
> final String selectQuery = "SELECT * FROM left_table;";
> final Configuration initialConfig = new Configuration();
> initialConfig.set(CoreOptions.DEFAULT_PARALLELISM, 1);
> final EnvironmentSettings initialSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(initialConfig)
> .build();
> final TableEnvironment initialTableEnv =
> TableEnvironment.create(initialSettings);
> // create job and consume two results
> initialTableEnv.executeSql(createTableQuery);
> final TableResult tableResult =
> initialTableEnv.sqlQuery(selectQuery).execute();
> tableResult.await();
> System.out.println(tableResultIterator.next());
> System.out.println(tableResultIterator.next());
> // stop job with savepoint
> final String savepointPath;
> try (CloseableIterator tableResultIterator =
> tableResult.collect()) {
> final JobClient jobClient =
>
> tableResult.getJobClient().orElseThrow(IllegalStateException::new);
> final File savepointDirectory = Files.createTempDir();
> savepointPath =
> jobClient
> .stopWithSavepoint(
> true,
> savepointDirectory.getAbsolutePath(),
> SavepointFormatType.CANONICAL)
> .get();
> }
> // restart the very same job from the savepoint
> final SavepointRestoreSettings savepointRestoreSettings =
> SavepointRestoreSettings.forPath(savepointPath, true);
> final Configuration restartConfig = new Configuration(initialConfig);
> SavepointRestoreSettings.toConfiguration(savepointRestoreSettings,
> restartConfig);
> final EnvironmentSettings restartSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(restartConfig)
> .build();
> final TableEnvironment restartTableEnv =
> TableEnvironment.create(restartSettings);
> restartTableEnv.executeSql(createTableQuery);
> restartTableEnv.sqlQuery(selectQuery).execute().print();
> }
> {code}
> h3. Expected behavior
> The job continues omitting the inital two records and starts printing results
> from 2 onwards.
> h3. Observed behavior
> No results are printed. The logs show that an invalid request was
[jira] [Commented] (FLINK-31143) Invalid request: offset doesn't match when restarting from a savepoint
[
https://issues.apache.org/jira/browse/FLINK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692463#comment-17692463
]
Caizhi Weng commented on FLINK-31143:
-
Hi [~mapohl] [~Weijie Guo].
When implementing {{collect}} I only considered job restarts and didn't
consider client restarts, so {{collect}} + savepoint is currently not supported.
However it isn't hard to implement because the collect sink has sent everything
needed to the client, just that client didn't handle the case. I can implement
this but I think this is actually a new feature rather than a bug fix. As we're
very close to releasing I think it might be more proper to introduce this
feature in 1.17.1. What do you think?
> Invalid request: offset doesn't match when restarting from a savepoint
> --
>
> Key: FLINK-31143
> URL: https://issues.apache.org/jira/browse/FLINK-31143
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Critical
>
> I tried to run the following case:
> {code:java}
> public static void main(String[] args) throws Exception {
> final String createTableQuery =
> "CREATE TABLE left_table (a int, c varchar) "
> + "WITH ("
> + " 'connector' = 'datagen', "
> + " 'rows-per-second' = '1', "
> + " 'fields.a.kind' = 'sequence', "
> + " 'fields.a.start' = '0', "
> + " 'fields.a.end' = '10'"
> + ");";
> final String selectQuery = "SELECT * FROM left_table;";
> final Configuration initialConfig = new Configuration();
> initialConfig.set(CoreOptions.DEFAULT_PARALLELISM, 1);
> final EnvironmentSettings initialSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(initialConfig)
> .build();
> final TableEnvironment initialTableEnv =
> TableEnvironment.create(initialSettings);
> // create job and consume two results
> initialTableEnv.executeSql(createTableQuery);
> final TableResult tableResult =
> initialTableEnv.sqlQuery(selectQuery).execute();
> tableResult.await();
> System.out.println(tableResultIterator.next());
> System.out.println(tableResultIterator.next());
> // stop job with savepoint
> final String savepointPath;
> try (CloseableIterator tableResultIterator =
> tableResult.collect()) {
> final JobClient jobClient =
>
> tableResult.getJobClient().orElseThrow(IllegalStateException::new);
> final File savepointDirectory = Files.createTempDir();
> savepointPath =
> jobClient
> .stopWithSavepoint(
> true,
> savepointDirectory.getAbsolutePath(),
> SavepointFormatType.CANONICAL)
> .get();
> }
> // restart the very same job from the savepoint
> final SavepointRestoreSettings savepointRestoreSettings =
> SavepointRestoreSettings.forPath(savepointPath, true);
> final Configuration restartConfig = new Configuration(initialConfig);
> SavepointRestoreSettings.toConfiguration(savepointRestoreSettings,
> restartConfig);
> final EnvironmentSettings restartSettings =
> EnvironmentSettings.newInstance()
> .inStreamingMode()
> .withConfiguration(restartConfig)
> .build();
> final TableEnvironment restartTableEnv =
> TableEnvironment.create(restartSettings);
> restartTableEnv.executeSql(createTableQuery);
> restartTableEnv.sqlQuery(selectQuery).execute().print();
> }
> {code}
> h3. Expected behavior
> The job continues omitting the inital two records and starts printing results
> from 2 onwards.
> h3. Observed behavior
> No results are printed. The logs show that an invalid request was handled:
> {code:java}
> org.apache.flink.streaming.api.operators.collect.CollectSinkFunction [] -
> Invalid request. Received version = b497a74f-e85c-404b-8df3-1b4b1a0c2de1,
> offset = 0, while expected version = b497a74f-e85c-404b-8df3-1b4b1a0c2de1,
> offset = 1
> {code}
> It looks like the right offset is not picked up from the savepoint (see
>
[jira] [Commented] (FLINK-31161) upgrade MojoHaus Versions Maven Plugin to 2.14.2
[
https://issues.apache.org/jira/browse/FLINK-31161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692462#comment-17692462
]
Ran Tao commented on FLINK-31161:
-
hi, guys. i got same issue. in my company, we have a extended project inherit
from flink, we can not use mvn versions:set to change version cause by
[ERROR] Failed to execute goal
org.codehaus.mojo:versions-maven-plugin:2.15.0:set (default-cli) on project
flink-plus: Project version is inherited from parent. -> [Help 1].
because current plugin can not change inherited parent version.
> upgrade MojoHaus Versions Maven Plugin to 2.14.2
>
>
> Key: FLINK-31161
> URL: https://issues.apache.org/jira/browse/FLINK-31161
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.18.0
>Reporter: jackylau
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
> Attachments: image-2023-02-21-12-17-20-195.png
>
>
> when we use multiple project, the parrent struct like this
> [https://stackoverflow.com/questions/39449275/update-parent-version-in-a-maven-projects-module.]
> when i use
> mvn org.codehaus.mojo:versions-maven-plugin:2.8.1:update-parent
> -DparentVersion=[1.15.2.1] -DallowSnapshots
> could not change parrent version.
> !image-2023-02-21-12-17-20-195.png!
>
> it is fixed added by skipResolution by upgrading to 2.14.2
> [https://www.mojohaus.org/versions/versions-maven-plugin/update-parent-mojo.html]
> {code:java}
> mvn org.codehaus.mojo:versions-maven-plugin:2.14.2:update-parent
> -DparentVersion=[1.15.2.1] -DallowSnapshots -DskipResolution{code}
> [https://github.com/mojohaus/versions/tree/2.14.2]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31182) CompiledPlan cannot deserialize BridgingSqlFunction with MissingTypeStrategy
[
https://issues.apache.org/jira/browse/FLINK-31182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692450#comment-17692450
]
Jane Chan edited comment on FLINK-31182 at 2/23/23 3:02 AM:
[~Sergey Nuyanzin] Yes, the check on the output type inference strategy was
introduced in FunctionCatalogTableOperator in FLINK-15487 for FLIP-65 a long
time ago.
I guess it worked well before FLIP-190 was introduced
because`LogicalUnnestRule` will infer the output type and convert UNNEST to
EXPLODE. (see [LogicalUnnestRule
L#99|https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/LogicalUnnestRule.scala#L99]).
As a result, although the BuiltinFunctionDefinition for UNNEST always has a
"MISSING" type inference, it does not affect the execution.
However, after CompiledPlan is introduced, RexNodeJsonDeserializer relies on
the OperatorTable to lookup functions, which always get a static placeholder
(i.e. MISSING) for UNNEST, and due to this check, the deserialization failed.
was (Author: qingyue):
[~Sergey Nuyanzin] Yes, the check on the output type inference strategy was
introduced in FunctionCatalogTableOperator in FLINK-15487 for FLIP-65 a long
time ago. I guess it worked well before FLIP-190 was introduced. The
RexNodeJsonDeserializer relies on the OperatorTable to lookup functions, and
due to this check, the deserialization failed.
> CompiledPlan cannot deserialize BridgingSqlFunction with MissingTypeStrategy
>
>
> Key: FLINK-31182
> URL: https://issues.apache.org/jira/browse/FLINK-31182
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.17.0, 1.18.0, 1.17.1
>Reporter: Jane Chan
>Priority: Major
> Attachments: screenshot-1.png
>
>
> This issue is reported from the [user mail
> list|https://lists.apache.org/thread/y6fgzyx330omhkr40376knw8k4oczz3s].
> The stacktrace is
> {code:java}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlCaused by: org.apache.flink.table.api.TableException: Could not resolve
> internal system function '$UNNEST_ROWS$1'. This is a bug, please file an
> issue.
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserializeInternalFunction(RexNodeJsonDeserializer.java:392)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserializeSqlOperator(RexNodeJsonDeserializer.java:337)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserializeCall(RexNodeJsonDeserializer.java:307)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserialize(RexNodeJsonDeserializer.java:146)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserialize(RexNodeJsonDeserializer.java:128)
> at
> org.apache.flink.table.planner.plan.nodes.exec.serde.RexNodeJsonDeserializer.deserialize(RexNodeJsonDeserializer.java:115)
> {code}
> The root cause is that although ModuleManager can resolve '$UNNEST_ROWS$1',
> the output type strategy is "Missing"; as a result,
> FunctionCatalogOperatorTable#convertToBridgingSqlFunction returns empty.
> !screenshot-1.png|width=675,height=295!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27115) Error while executing BLOB connection. java.io.IOException: Unknown operation 80
[ https://issues.apache.org/jira/browse/FLINK-27115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17692460#comment-17692460 ] Xintong Song commented on FLINK-27115: -- I cannot see the screenshot. Is this exception causing problems like job failure or it's just a warning / error in logs? bq. this is mostly because there are requests outside of Flink to the BlobServer That sounds like a malicious attack. While logging the requesting address may not be expensive, I'm not entirely sure we should introduce changes only for investigating mis-usages in some particular scenarios. Do we need this log in other common scenarios where such undesired requests do not exist? > Error while executing BLOB connection. java.io.IOException: Unknown operation > 80 > > > Key: FLINK-27115 > URL: https://issues.apache.org/jira/browse/FLINK-27115 > Project: Flink > Issue Type: Bug >Affects Versions: 1.12.7 >Reporter: zhiyuan >Priority: Major > > hi, > I have a Flink SQL job running online. Every morning, I will report the > following screenshot error. I have enabled the debug log to check and find > nothing > version 1.12 > !https://attachment.outlook.live.net/owa/MSA%3Azhiyuan.franklin%40outlook.com/service.svc/s/GetAttachmentThumbnail?id=AQMkADAwATM0MDAAMS0zYzc1LTFmNGQtMDACLTAwCgBGAAAD39WjdeaLP0WTyi0RfpiKawcAZU0SuCA2lEmxEZ%2BxAo0sZwAAAgEJZU0SuCA2lEmxEZ%2BxAo0sZwAAAFJWGd8BEgAQAAXCfQrjuKVKrok%2FasxKyoI%3D=2=1=eyJhbGciOiJSUzI1NiIsImtpZCI6IkZBRDY1NDI2MkM2QUYyOTYxQUExRThDQUI3OEZGMUIyNzBFNzA3RTkiLCJ0eXAiOiJKV1QiLCJ4NXQiOiItdFpVSml4cThwWWFvZWpLdDRfeHNuRG5CLWsifQ.eyJvcmlnaW4iOiJodHRwczovL291dGxvb2subGl2ZS5jb20iLCJ1YyI6IjFiNmM2MGU3YWFlYTQ5M2E4NzViZjY4NjQwNTI4MzA5IiwidmVyIjoiRXhjaGFuZ2UuQ2FsbGJhY2suVjEiLCJhcHBjdHhzZW5kZXIiOiJPd2FEb3dubG9hZEA4NGRmOWU3Zi1lOWY2LTQwYWYtYjQzNS1hYWFhYWFhYWFhYWEiLCJpc3NyaW5nIjoiV1ciLCJhcHBjdHgiOiJ7XCJtc2V4Y2hwcm90XCI6XCJvd2FcIixcInB1aWRcIjpcIjkxNDc5ODk4MzU4NTYxM1wiLFwic2NvcGVcIjpcIk93YURvd25sb2FkXCIsXCJvaWRcIjpcIjAwMDM0MDAxLTNjNzUtMWY0ZC0wMDAwLTAwMDAwMDAwMDAwMFwiLFwicHJpbWFyeXNpZFwiOlwiUy0xLTI4MjctMjEyOTkzLTEwMTQzMDg2ODVcIn0iLCJuYmYiOjE2NDkzMjQ5ODIsImV4cCI6MTY0OTMyNTU4MiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQDg0ZGY5ZTdmLWU5ZjYtNDBhZi1iNDM1LWFhYWFhYWFhYWFhYSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC9hdHRhY2htZW50Lm91dGxvb2subGl2ZS5uZXRAODRkZjllN2YtZTlmNi00MGFmLWI0MzUtYWFhYWFhYWFhYWFhIiwiaGFwcCI6Im93YSJ9.cXk2gaCUX42hpan8Y6m_xMFXUuwap_WOItLw2Cnt2ZJUx01w1Lnlt-JICWKz18mNjUgsCliu1ds3o0AIs4etAwxHFjFwzIQ1Qza7Q_RABlPl9twLkW1ukQYppHvgCeSZQZ4r3-OfsOXeKVXexui19_tAW5iUKhOJihCkU3B84hvX5DGdkMo1daQP9-89fGT4JfoAxSmnNhTjkZ7THULYgz6sHumrOh82Nd_Qqz7VE5jv3tkZw-qCFgyUhHTX1kA51W_GBcgjIw85mJC7iBwnu-mh5ix2Vgawh9aNGcZTDS4FOvSdcU20Nci_BsPsCsaECdv1UCEJnYDSDD73ySWmSg=N_cwUTbUw0-7mNK6xvPrRTBKApR8GNoYAgYgmvWw1rWGZX3VKlEy8VAzuIMf5IJJxzBTtG1-q4o.=outlook.live.com=20220401003.04=true! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
