[GitHub] [flink] flinkbot commented on pull request #23448: [FLINK-33050][table] Atomicity is not supported prompting the user to disable
flinkbot commented on PR #23448: URL: https://github.com/apache/flink/pull/23448#issuecomment-1728715171 ## CI report: * 170ad01d722059d47c254ee52cf170501d6cbfdc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Tartarus0zm opened a new pull request, #23448: [FLINK-33050][table] Atomicity is not supported prompting the user to disable
Tartarus0zm opened a new pull request, #23448: URL: https://github.com/apache/flink/pull/23448 ## What is the purpose of the change * When atomicity is enabled, an exception may occur when creating a DynamicTableSink, and we need to prompt the user to disable atomicity. * When we use InMemoryCatalog, RTAS drop table will only delete the metadata, not clean up the underlying data files, RTAS write data does not use overwrite semantics by default, so it looks like the data is duplicated, this problem needs to be clarified in the documentation. ## Brief change log * add try-catch * add docs ## Verifying this change no ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-32938) flink-connector-pulsar should remove all `PulsarAdmin` calls
[
https://issues.apache.org/jira/browse/FLINK-32938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen resolved FLINK-32938.
---
Fix Version/s: pulsar-4.1.0
Resolution: Fixed
master via 78d00ea9e3e278d4ce2fbb0c8a8d380abef7b858
> flink-connector-pulsar should remove all `PulsarAdmin` calls
>
>
> Key: FLINK-32938
> URL: https://issues.apache.org/jira/browse/FLINK-32938
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Reporter: Neng Lu
>Assignee: Neng Lu
>Priority: Major
> Labels: pull-request-available
> Fix For: pulsar-4.1.0
>
>
> The flink-connector-pulsar should not access and interact with the admin
> endpoint. This could introduce potential security issues.
> In a production environment, a Pulsar cluster admin will not grant the
> permissions for the flink application to conduct any admin operations.
> Currently, the connector does various admin calls:
> ```{{{}{}}}{{{}{}}}
> PulsarAdmin.topics().getPartitionedTopicMetadata(topic)
> PulsarAdmin.namespaces().getTopics(namespace)
> PulsarAdmin.topics().getLastMessageId(topic)
> PulsarAdmin.topics().getMessageIdByTimestamp(topic, timestamp)
> PulsarAdmin.topics().getSubscriptions(topic)
> PulsarAdmin.topics().createSubscription(topic, subscription,
> MessageId.earliest)
> PulsarAdmin.topics().resetCursor(topic, subscription, initial, !include)
> ```
> We need to replace these calls with consumer or client calls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-32938) flink-connector-pulsar should remove all `PulsarAdmin` calls
[
https://issues.apache.org/jira/browse/FLINK-32938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zili Chen reassigned FLINK-32938:
-
Assignee: Neng Lu
> flink-connector-pulsar should remove all `PulsarAdmin` calls
>
>
> Key: FLINK-32938
> URL: https://issues.apache.org/jira/browse/FLINK-32938
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Pulsar
>Reporter: Neng Lu
>Assignee: Neng Lu
>Priority: Major
> Labels: pull-request-available
>
> The flink-connector-pulsar should not access and interact with the admin
> endpoint. This could introduce potential security issues.
> In a production environment, a Pulsar cluster admin will not grant the
> permissions for the flink application to conduct any admin operations.
> Currently, the connector does various admin calls:
> ```{{{}{}}}{{{}{}}}
> PulsarAdmin.topics().getPartitionedTopicMetadata(topic)
> PulsarAdmin.namespaces().getTopics(namespace)
> PulsarAdmin.topics().getLastMessageId(topic)
> PulsarAdmin.topics().getMessageIdByTimestamp(topic, timestamp)
> PulsarAdmin.topics().getSubscriptions(topic)
> PulsarAdmin.topics().createSubscription(topic, subscription,
> MessageId.earliest)
> PulsarAdmin.topics().resetCursor(topic, subscription, initial, !include)
> ```
> We need to replace these calls with consumer or client calls.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33127) HeapKeyedStateBackend: use buffered I/O to speed up local recovery
[ https://issues.apache.org/jira/browse/FLINK-33127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanfei Lei reassigned FLINK-33127: -- Assignee: Yangyang ZHANG > HeapKeyedStateBackend: use buffered I/O to speed up local recovery > -- > > Key: FLINK-33127 > URL: https://issues.apache.org/jira/browse/FLINK-33127 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Yangyang ZHANG >Assignee: Yangyang ZHANG >Priority: Major > Attachments: thread_dump.png > > > Recently, I observed a slow restore case in local recovery using hashmap > statebackend. > It took 147 seconds to restore from a 467MB snapshot, 9 times slower than > that (16s) when restore from remote fs. > The thread dump show that It read local snapshot file directly by unbuffered > FileInputStream / fs.local.LocalDataInputStream. > !thread_dump.png! > Maybe we can wrap with BufferInputStream to speed up local recovery. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33127) HeapKeyedStateBackend: use buffered I/O to speed up local recovery
Yangyang ZHANG created FLINK-33127: -- Summary: HeapKeyedStateBackend: use buffered I/O to speed up local recovery Key: FLINK-33127 URL: https://issues.apache.org/jira/browse/FLINK-33127 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Reporter: Yangyang ZHANG Attachments: thread_dump.png Recently, I observed a slow restore case in local recovery using hashmap statebackend. It took 147 seconds to restore from a 467MB snapshot, 9 times slower than that (16s) when restore from remote fs. The thread dump show that It read local snapshot file directly by unbuffered FileInputStream / fs.local.LocalDataInputStream. !thread_dump.png! Maybe we can wrap with BufferInputStream to speed up local recovery. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33126) Fix EventTimeAllWindowCheckpointingITCase jobName typo
[ https://issues.apache.org/jira/browse/FLINK-33126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33126: --- Labels: pull-request-available (was: ) > Fix EventTimeAllWindowCheckpointingITCase jobName typo > -- > > Key: FLINK-33126 > URL: https://issues.apache.org/jira/browse/FLINK-33126 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.1 >Reporter: Yue Ma >Assignee: Yue Ma >Priority: Minor > Labels: pull-request-available > > Fix EventTimeAllWindowCheckpointingITCase jobName Typo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] mayuehappy commented on pull request #23444: [FLINK-33126] Fix EventTimeAllWindowCheckpointingITCase jobName typo
mayuehappy commented on PR #23444: URL: https://github.com/apache/flink/pull/23444#issuecomment-1728690076 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33060) Fix the javadoc of ListState.update/addAll about not allowing null value
[ https://issues.apache.org/jira/browse/FLINK-33060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33060: --- Labels: pull-request-available (was: ) > Fix the javadoc of ListState.update/addAll about not allowing null value > > > Key: FLINK-33060 > URL: https://issues.apache.org/jira/browse/FLINK-33060 > Project: Flink > Issue Type: Bug >Reporter: Zakelly Lan >Assignee: Zakelly Lan >Priority: Minor > Labels: pull-request-available > > After FLINK-8411, the ListState.update/add/addAll do not allow a null value > passed in, while the javadoc says "If null is passed in, the state value will > remain unchanged". This should be fixed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] elkhand commented on a diff in pull request #23406: [FLINK-32884] [flink-clients] PyFlink remote execution should support URLs with paths and https scheme
elkhand commented on code in PR #23406:
URL: https://github.com/apache/flink/pull/23406#discussion_r1332375725
##
flink-clients/src/main/java/org/apache/flink/client/program/rest/RestClusterClient.java:
##
@@ -255,10 +266,22 @@ private RestClusterClient(
RestClusterClientConfiguration.fromConfiguration(configuration);
this.tempDir = tempDir;
+this.customHttpHeaders =
+ClientUtils.readHeadersFromEnvironmentVariable(
+ConfigConstants.FLINK_REST_CLIENT_HEADERS);
+jobmanagerUrl =
+new URL(
+
configuration.getBoolean(SecurityOptions.SSL_REST_ENABLED)
+? "https"
+: "http",
+configuration.getString(JobManagerOptions.ADDRESS),
+configuration.getInteger(JobManagerOptions.PORT),
+configuration.getString(RestOptions.PATH));
+
if (restClient != null) {
this.restClient = restClient;
} else {
-this.restClient = new RestClient(configuration, executorService);
+this.restClient = RestClient.forUrl(configuration,
executorService, jobmanagerUrl);
Review Comment:
Thanks @gaborgsomogyi .
Added tests for jobmanagerUrl and customHeaders.
cc: @mxm .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-32974) RestClusterClient always leaks flink-rest-client-jobgraphs* directories
[
https://issues.apache.org/jira/browse/FLINK-32974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767357#comment-17767357
]
Lijie Wang commented on FLINK-32974:
Fix via
master: 6034d5ff335dc970672c290851811235399452fb
release-1.18: 2aeb99804ba56c008df0a1730f3246d3fea856b9
release-1.17: 7c9e05ea8c67b12c657b60cd5e6d1bea52b4f9a3
> RestClusterClient always leaks flink-rest-client-jobgraphs* directories
> ---
>
> Key: FLINK-32974
> URL: https://issues.apache.org/jira/browse/FLINK-32974
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.18.0, 1.17.2
>Reporter: Lijie Wang
>Assignee: Lijie Wang
>Priority: Critical
>
> After FLINK-32226, a temporary directory(named
> {{flink-rest-client-jobgraphs*}}) is created when creating a new
> RestClusterClient, but this directory will never be cleaned up.
> This will result in a lot of {{flink-rest-client-jobgraphs*}} directories
> under {{/tmp}}, especially when using
> CollectDynamicSink/CollectResultFetcher, which may cause the inode to be used
> up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-32974) RestClusterClient always leaks flink-rest-client-jobgraphs* directories
[
https://issues.apache.org/jira/browse/FLINK-32974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lijie Wang resolved FLINK-32974.
Fix Version/s: 1.18.0
1.17.2
Resolution: Fixed
> RestClusterClient always leaks flink-rest-client-jobgraphs* directories
> ---
>
> Key: FLINK-32974
> URL: https://issues.apache.org/jira/browse/FLINK-32974
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.18.0, 1.17.2
>Reporter: Lijie Wang
>Assignee: Lijie Wang
>Priority: Critical
> Fix For: 1.18.0, 1.17.2
>
>
> After FLINK-32226, a temporary directory(named
> {{flink-rest-client-jobgraphs*}}) is created when creating a new
> RestClusterClient, but this directory will never be cleaned up.
> This will result in a lot of {{flink-rest-client-jobgraphs*}} directories
> under {{/tmp}}, especially when using
> CollectDynamicSink/CollectResultFetcher, which may cause the inode to be used
> up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32405) Initialize catalog listener for CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-32405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767356#comment-17767356 ] Benchao Li commented on FLINK-32405: [~zjureel] Thanks for the updating, I've also added the link. > Initialize catalog listener for CatalogManager > -- > > Key: FLINK-32405 > URL: https://issues.apache.org/jira/browse/FLINK-32405 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: FangYong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32405) Initialize catalog listener for CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-32405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fang Yong closed FLINK-32405. - Resolution: Duplicate Duplicate with FLINK-32404 > Initialize catalog listener for CatalogManager > -- > > Key: FLINK-32405 > URL: https://issues.apache.org/jira/browse/FLINK-32405 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: FangYong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32405) Initialize catalog listener for CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-32405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767354#comment-17767354 ] Fang Yong commented on FLINK-32405: --- OK, it is a duplicate issue with FLINK-32404 > Initialize catalog listener for CatalogManager > -- > > Key: FLINK-32405 > URL: https://issues.apache.org/jira/browse/FLINK-32405 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: FangYong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-32405) Initialize catalog listener for CatalogManager
[ https://issues.apache.org/jira/browse/FLINK-32405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fang Yong reopened FLINK-32405: --- > Initialize catalog listener for CatalogManager > -- > > Key: FLINK-32405 > URL: https://issues.apache.org/jira/browse/FLINK-32405 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.18.0 >Reporter: Fang Yong >Assignee: FangYong >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage
[ https://issues.apache.org/jira/browse/FLINK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-33044: --- Fix Version/s: 1.18.0 > Reduce the frequency of triggering flush for the disk tier of the tiered > storage > > > Key: FLINK-33044 > URL: https://issues.apache.org/jira/browse/FLINK-33044 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0, 1.19.0 > > > The disk cache of tiered storage will flush at the end of each subpartition's > segment, which is too frequent and is bad for performance. We should improve > it with some better flushing methods, e.g. flushing buffers with batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage
[ https://issues.apache.org/jira/browse/FLINK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo closed FLINK-33044. -- Resolution: Fixed > Reduce the frequency of triggering flush for the disk tier of the tiered > storage > > > Key: FLINK-33044 > URL: https://issues.apache.org/jira/browse/FLINK-33044 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0, 1.19.0 > > > The disk cache of tiered storage will flush at the end of each subpartition's > segment, which is too frequent and is bad for performance. We should improve > it with some better flushing methods, e.g. flushing buffers with batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage
[ https://issues.apache.org/jira/browse/FLINK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767089#comment-17767089 ] Weijie Guo edited comment on FLINK-33044 at 9/21/23 2:04 AM: - master(1.19) via b076c52d8da914e81c3e004c0b0c7883463bb151. release-1.18 via 40bead52d200f8aa9cd30336dcc4c3f6f8c68636. was (Author: weijie guo): master(1.19) via b076c52d8da914e81c3e004c0b0c7883463bb151. > Reduce the frequency of triggering flush for the disk tier of the tiered > storage > > > Key: FLINK-33044 > URL: https://issues.apache.org/jira/browse/FLINK-33044 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > The disk cache of tiered storage will flush at the end of each subpartition's > segment, which is too frequent and is bad for performance. We should improve > it with some better flushing methods, e.g. flushing buffers with batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage
[ https://issues.apache.org/jira/browse/FLINK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33044: --- Labels: pull-request-available (was: ) > Reduce the frequency of triggering flush for the disk tier of the tiered > storage > > > Key: FLINK-33044 > URL: https://issues.apache.org/jira/browse/FLINK-33044 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > The disk cache of tiered storage will flush at the end of each subpartition's > segment, which is too frequent and is bad for performance. We should improve > it with some better flushing methods, e.g. flushing buffers with batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] TanYuxin-tyx commented on pull request #23442: [BP][FLINK-33044][network] Reduce the frequency of triggering flush for the disk tier of the tiered storage
TanYuxin-tyx commented on PR #23442: URL: https://github.com/apache/flink/pull/23442#issuecomment-1728657634 Thanks @reswqa for reviewing the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa merged pull request #23442: [BP][FLINK-33044][network] Reduce the frequency of triggering flush for the disk tier of the tiered storage
reswqa merged PR #23442: URL: https://github.com/apache/flink/pull/23442 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] luoyuxia closed pull request #23389: [FLINK-33050][table] Atomicity is not supported prompting the user to disable
luoyuxia closed pull request #23389: [FLINK-33050][table] Atomicity is not supported prompting the user to disable URL: https://github.com/apache/flink/pull/23389 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-32848) [JUnit5 Migration] The persistence, query, registration, rpc and shuffle packages of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fang Yong closed FLINK-32848. - Resolution: Fixed Fixed by 2ced46b6ed4ebfb20f9b392cba4e789e16e4da7d...f2cb1d247283344e9194e63931a2948e09f73c93 > [JUnit5 Migration] The persistence, query, registration, rpc and shuffle > packages of flink-runtime module > - > > Key: FLINK-32848 > URL: https://issues.apache.org/jira/browse/FLINK-32848 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Zhanghao Chen >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] becketqin commented on a diff in pull request #23313: [FLINK-20767][table planner] Support filter push down on nested fields
becketqin commented on code in PR #23313:
URL: https://github.com/apache/flink/pull/23313#discussion_r1332312205
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java:
##
@@ -1183,7 +1194,9 @@ private Map, Collection>
filterAllData(
for (Row row : allData.get(partition)) {
boolean isRetained =
FilterUtils.isRetainedAfterApplyingFilterPredicates(
-filterPredicates, getValueGetter(row));
+filterPredicates,
+getValueGetter(row),
+Optional.of(getNestedValueGetter(row)));
Review Comment:
I think this is more of a programing convention for language. Optional is
equivalent to `@Nullable` annotation in Java.
The Optional parameter is kind of a paradox. If a parameter is Optional,
that means a null value is acceptable. So users can actually pass the nullable
value without wrapping it with Optional. If a parameter is not Optional, it
implies that the value cannot be null, so user need to make a null check on the
value. If it is null, likely an exception needs to be thrown, which is pretty
much what the invoked method will do when it sees null for the non-optional
value. So in either case, having an Optional parameter does not bring much
value, while introduce extra wrapping in the first case.
From a caller's perspective, a parameter value is known to be null or not,
if the value is going to be passed to the method call anyways, wrapping the
value with Optional is extra burden compared with passing the value directly.
On the other handle, the return value of the method call is unknown to the
caller. If the return value of a method call is an Optional, it explicitly
tells the caller that the return value needs to be checked because it might
return nothing. This is why Optional is useful for return value, but not the
parameters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-33018) GCP Pubsub PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream failed
[
https://issues.apache.org/jira/browse/FLINK-33018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33018:
---
Labels: auto-deprioritized-blocker pull-request-available (was:
pull-request-available stale-blocker)
Priority: Critical (was: Blocker)
This issue was labeled "stale-blocker" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually a Blocker,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> GCP Pubsub
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream
> failed
>
>
> Key: FLINK-33018
> URL: https://issues.apache.org/jira/browse/FLINK-33018
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub
>Affects Versions: gcp-pubsub-3.0.2
>Reporter: Martijn Visser
>Priority: Critical
> Labels: auto-deprioritized-blocker, pull-request-available
>
> https://github.com/apache/flink-connector-gcp-pubsub/actions/runs/6061318336/job/16446392844#step:13:507
> {code:java}
> [INFO]
> [INFO] Results:
> [INFO]
> Error: Failures:
> Error:
> PubSubConsumingTest.testStoppingConnectorWhenDeserializationSchemaIndicatesEndOfStream:119
>
> expected: ["1", "2", "3"]
> but was: ["1", "2"]
> [INFO]
> Error: Tests run: 30, Failures: 1, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-32223: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32418) ClassNotFoundException when using flink-protobuf with sql-client
[
https://issues.apache.org/jira/browse/FLINK-32418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32418:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 30 days, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it.
> ClassNotFoundException when using flink-protobuf with sql-client
>
>
> Key: FLINK-32418
> URL: https://issues.apache.org/jira/browse/FLINK-32418
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.16.2
>Reporter: Michael Kreis
>Assignee: Michael Kreis
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: full-stacktrace.log
>
>
> When the protobuf format in the kafka connector is used via the sql-client it
> is not able to load the generated protobuf classes which are either passed
> via `-j /protobuf-classes.jar` or added in the script via ADD JAR
> '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded
> but when the protobuf classes are loaded a ClassNotFoundException occurs.
> executed command:
> {code:java}
> sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
> {code}
> protobuf-table.sql
> {code:sql}
> ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
> ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
> SHOW JARS;
> CREATE TABLE POSITIONS(id BIGINT) WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'topic' = 'protbuf-topic',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'flink-protobuf',
> 'properties.security.protocol' = 'SASL_PLAINTEXT',
> 'properties.sasl.mechanism' = 'SCRAM-SHA-512',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.scram.ScramLoginModule required
> username="user" password="";',
> 'scan.startup.mode' = 'earliest-offset',
> 'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
> 'protobuf.ignore-parse-errors' = 'true'
> );
> SELECT * FROM POSITIONS;
> {code}
> exception in the log:
> {code:java}
> Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown
> Source)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
> Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Unknown Source)
> at
> org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
> ... 36 more
> {code}
> This also seems somehow related to FLINK-30318
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30649) Shutting down MiniCluster times out
[
https://issues.apache.org/jira/browse/FLINK-30649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767290#comment-17767290
]
Matthias Pohl commented on FLINK-30649:
---
Thanks for volunteering. Do you have an idea on how to proceed here?
> Shutting down MiniCluster times out
> ---
>
> Key: FLINK-30649
> URL: https://issues.apache.org/jira/browse/FLINK-30649
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Test Infrastructure
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: stale-assigned, starter, test-stability
>
> {{Run kubernetes session test (default input)}} failed with a timeout.
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44748=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14=6317]
> It appears that there was some issue with shutting down the pods of the
> MiniCluster:
> {code:java}
> 2023-01-12T08:22:13.1388597Z timed out waiting for the condition on
> pods/flink-native-k8s-session-1-7dc9976688-gq788
> 2023-01-12T08:22:13.1390040Z timed out waiting for the condition on
> pods/flink-native-k8s-session-1-taskmanager-1-1 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-15736) Support Java 17 (LTS)
[ https://issues.apache.org/jira/browse/FLINK-15736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767273#comment-17767273 ] Chesnay Schepler commented on FLINK-15736: -- Documentation added: master: 388601f1b75abd443e149aaa4584d366072a1b0e 1.18: 9c1318ca7fa5b2e7b11827068ad1288483aaa464 > Support Java 17 (LTS) > - > > Key: FLINK-15736 > URL: https://issues.apache.org/jira/browse/FLINK-15736 > Project: Flink > Issue Type: New Feature > Components: Build System >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: auto-deprioritized-major, pull-request-available, > stale-assigned > Fix For: 1.18.0 > > > Long-term issue for preparing Flink for Java 17. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol merged pull request #23445: [FLINK-15736][docs] Add Java compatibility page
zentol merged PR #23445: URL: https://github.com/apache/flink/pull/23445 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] tzulitai commented on a diff in pull request #20: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness
tzulitai commented on code in PR #20: URL: https://github.com/apache/flink-connector-kafka/pull/20#discussion_r1332045177 ## flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSource.java: ## @@ -98,20 +98,24 @@ private final KafkaRecordDeserializationSchema deserializationSchema; // The configurations. private final Properties props; +// Client rackId callback +private final Supplier rackIdSupplier; Review Comment: I think we must address this before we merge this PR. At the very least, the `Supplier` needs to be serializable. You can use the `SerializableSupplier` interface from Flink. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33104) Nightly run for Flink Kafka connector fails due to architecture tests failing
[
https://issues.apache.org/jira/browse/FLINK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767232#comment-17767232
]
Mason Chen commented on FLINK-33104:
Do we want to ignore the violations for now? or do we want to migrate tests to
JUnit5? I can help do some of the refactoring.
Moreover, it seems like any arch unit rule addition can easily break builds. I
don't know if there is anything to make this situation more stable since we
rely on the SNAPSHOT artifacts
> Nightly run for Flink Kafka connector fails due to architecture tests failing
> -
>
> Key: FLINK-33104
> URL: https://issues.apache.org/jira/browse/FLINK-33104
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: kafka-3.1.0
>Reporter: Martijn Visser
>Priority: Blocker
>
> {code:java}
> 2023-09-17T00:29:07.1675694Z [WARNING] Tests run: 18, Failures: 0, Errors: 0,
> Skipped: 9, Time elapsed: 308.532 s - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerMigrationTest
> 2023-09-17T00:29:07.5171608Z [INFO]
> 2023-09-17T00:29:07.5172360Z [INFO] Results:
> 2023-09-17T00:29:07.5172773Z [INFO]
> 2023-09-17T00:29:07.5173139Z [ERROR] Failures:
> 2023-09-17T00:29:07.5174181Z [ERROR] Architecture Violation [Priority:
> MEDIUM] - Rule 'ITCASE tests should use a MiniCluster resource or extension'
> was violated (13 times):
> 2023-09-17T00:29:07.5176050Z
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase does
> not satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5177452Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5179831Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5181277Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5182154Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5182951Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5183906Z
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase does not satisfy: only
> one of the following predicates match:
> 2023-09-17T00:29:07.5184769Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5185812Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5186880Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5187929Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5189073Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5190076Z
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase does not
> satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5190946Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5191983Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5192845Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
>
[jira] [Commented] (FLINK-33121) Failed precondition in JobExceptionsHandler due to concurrent global failures
[
https://issues.apache.org/jira/browse/FLINK-33121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767214#comment-17767214
]
Chesnay Schepler commented on FLINK-33121:
--
Had an offline chat; our suspicion is that something is calling the failure
handling logic of the scheduler without running this call in the main thread.
We'll try to confirm this theory.
> Failed precondition in JobExceptionsHandler due to concurrent global failures
> -
>
> Key: FLINK-33121
> URL: https://issues.apache.org/jira/browse/FLINK-33121
> Project: Flink
> Issue Type: Bug
>Reporter: Panagiotis Garefalakis
>Priority: Major
>
> {{JobExceptionsHandler#createRootExceptionInfo}} *only* allows concurrent
> exceptions that are local failures *--* otherwise throws an assertion as part
> of {{{}asserLocalExceptionInfo{}}}.
> However, there are rare cases where multiple concurrent global failures are
> triggered and added to the failureCollection, before transitioning the job
> state to Failed e.g., through {{StateWithExecutionGraph#handleGlobalFailure}}
> of the AdaptiveScheduler.
> In this case the last added will be the root and the next one will trigger
> the assertion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23447: [FLINK-31650][metrics][rest] Cleanup terminal subtask attempt metrics to avoid confusion caused by outdated metrics
flinkbot commented on PR #23447: URL: https://github.com/apache/flink/pull/23447#issuecomment-1728104779 ## CI report: * a3d22bcb204a26716c51f71ef66c0cf58e5fff92 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31650) Incorrect busyMsTimePerSecond metric value for FINISHED task
[ https://issues.apache.org/jira/browse/FLINK-31650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767191#comment-17767191 ] Zhanghao Chen commented on FLINK-31650: --- [~wanglijie] [~JunRuiLi] Hi, I've created a PR that solves this issue by cleaning up terminal subtask attempt metrics and FINISHED tasks will show N/A for busy/backpressured on UI now. Could you help review it when you are free? > Incorrect busyMsTimePerSecond metric value for FINISHED task > > > Key: FLINK-31650 > URL: https://issues.apache.org/jira/browse/FLINK-31650 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics, Runtime / REST >Affects Versions: 1.17.0, 1.16.1, 1.15.4 >Reporter: Lijie Wang >Assignee: Zhanghao Chen >Priority: Major > Labels: pull-request-available > Attachments: busyMsTimePerSecond.png > > > As shown in the figure below, the busyMsTimePerSecond of the FINISHED task is > 100%, which is obviously unreasonable. > !busyMsTimePerSecond.png|width=1048,height=432! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31650) Incorrect busyMsTimePerSecond metric value for FINISHED task
[ https://issues.apache.org/jira/browse/FLINK-31650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31650: --- Labels: pull-request-available (was: ) > Incorrect busyMsTimePerSecond metric value for FINISHED task > > > Key: FLINK-31650 > URL: https://issues.apache.org/jira/browse/FLINK-31650 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics, Runtime / REST >Affects Versions: 1.17.0, 1.16.1, 1.15.4 >Reporter: Lijie Wang >Assignee: Zhanghao Chen >Priority: Major > Labels: pull-request-available > Attachments: busyMsTimePerSecond.png > > > As shown in the figure below, the busyMsTimePerSecond of the FINISHED task is > 100%, which is obviously unreasonable. > !busyMsTimePerSecond.png|width=1048,height=432! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] X-czh opened a new pull request, #23447: [FLINK-31650][metrics][rest] Cleanup terminal subtask attempt metrics to avoid confusion caused by outdated metrics
X-czh opened a new pull request, #23447: URL: https://github.com/apache/flink/pull/23447 ## What is the purpose of the change This pull request cleanups terminal subtask attempt metrics to avoid confusion caused by outdated metrics. For example, a FINISHED task may have its last updated 100 % busy time metrics retained in the metric store and shown on th UI, which is obviously unreasonable. ## Brief change log Removes terminal subtask attempt from the current execution attempts in `JobDetails` so that its metrics will be removed on update. ## Verifying this change - Added UT to test that task metrics are removed when the task is removed from current execution attempts map (this cantake place when the all subtasks reached terminal state) - Tested with a real job with a bounded source and verified on UI that the metrics were unavailable after the bounded source finished:  ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] venkata91 commented on a diff in pull request #23313: [FLINK-20767][table planner] Support filter push down on nested fields
venkata91 commented on code in PR #23313:
URL: https://github.com/apache/flink/pull/23313#discussion_r1331904794
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java:
##
@@ -1183,7 +1194,9 @@ private Map, Collection>
filterAllData(
for (Row row : allData.get(partition)) {
boolean isRetained =
FilterUtils.isRetainedAfterApplyingFilterPredicates(
-filterPredicates, getValueGetter(row));
+filterPredicates,
+getValueGetter(row),
+Optional.of(getNestedValueGetter(row)));
Review Comment:
I see the point around not passing `Optional` as method param. Let me try to
fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] venkata91 commented on a diff in pull request #23313: [FLINK-20767][table planner] Support filter push down on nested fields
venkata91 commented on code in PR #23313:
URL: https://github.com/apache/flink/pull/23313#discussion_r1331904794
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java:
##
@@ -1183,7 +1194,9 @@ private Map, Collection>
filterAllData(
for (Row row : allData.get(partition)) {
boolean isRetained =
FilterUtils.isRetainedAfterApplyingFilterPredicates(
-filterPredicates, getValueGetter(row));
+filterPredicates,
+getValueGetter(row),
+Optional.of(getNestedValueGetter(row)));
Review Comment:
I see the point around not passing `Optional` as method param. Let me try to
fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-32976) NullPointException when starting flink cluster in standalone mode
[
https://issues.apache.org/jira/browse/FLINK-32976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-32976:
---
Labels: pull-request-available (was: )
> NullPointException when starting flink cluster in standalone mode
> -
>
> Key: FLINK-32976
> URL: https://issues.apache.org/jira/browse/FLINK-32976
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.17.1
>Reporter: Feng Jin
>Assignee: Feng Jin
>Priority: Major
> Labels: pull-request-available
>
> It can be reproduced when starting flink cluster with hadoop configuration.
>
> {code:java}
> //代码占位符
> // Set up hadoop conf , hadoop classpath
> // start jobManager
> ./jobmanager.sh start-foreground {code}
>
> The error message as follows:
>
> {code:java}
> //代码占位符
> Caused by: java.ang.NullPointerException
> at org.apache.flink. runtime.
> security.token.hadoop.HadoopFSDelegationTokenProvider.getFileSystemsToAccess(HadoopFSDelegationTokenProvider.java:173)~[flink-dist-1.17.1.jar:1.17.1]
> at
> org.apache.flink.runtime.security.token.hadoop.HadoopFSDelegationTokenProvidertionTokens$1(HadoopFSDelegationTokenProvider.java:113)
> ~[flink-dist-1.17.1.jar:1.17.1
> at java.security.AccessController.doprivileged(Native Method)~[?:1.8.0 281]
> at javax.security.auth.Subject.doAs(Subject.java:422)~[?:1.8.0 281]
> at org. apache.hadoop .
> security.UserGroupInformation.doAs(UserGroupInformation. java:1876)
> ~flink-shacd-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar:3.1.1.7.2.1.0-327-9.0]
> at org. apache.flink. runtime.security.token .hadoop
> .HadoopFSDelegationTokenProvider.obtainDelegationTcens(HadoopFSDelegationTokenProvider.java:108)~flink-dist-1.17.1.jar:1.17.1]
> at org.apache.flink. runtime. security.token.DefaultDelegationTokenManager .
> lambda$obtainDel
> SAndGetNextRenewal$1(DefaultDelegationTokenManager .java:264)~
> flink-dist-1.17.1.jar:1.17.1]
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> ~?:1.8.0 281
> at
> java.util.HashMap$ValueSpliterator.forEachRemaining(HashMap.java:1628)~[?:1.8.0
> 281]at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)~?:1.8.0
> 281]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> ~?:1.8.0 281at
> java,util.stream.Reduce0ps$Reduce0p.evaluateSequential(Reduce0ps.java:708)~?:1.8.0
> 281]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)~[?:1.8.0
> 281]at java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:479)
> ~?:1.8.0 281
> at java.util.stream.ReferencePipeline.min(ReferencePipeline.java:520)~?:1.8.0
> 281at org. apache. flink. runtime.
> security.token.DefaultDelegationTokenManager
> .obtainDelegationTokensAndGeNextRenewal(DefaultDelegationTokenManager
> .java:286)~[flink-dist-1.17.1.jar:1.17.1
> at org.apache. flink.runtime. security.token.DefaultDelegationTokenManager.
> obtainDelegationTokens(DefaltDelegationTokenManager.java:242)~[flink-dist-1.17.1.jar:1.17.1]
> at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializes@)
> ~[flink-dist-1.17.1.jar:1.17.1]
> at
> org.apache.flink.runtime.entrypoint.clusterEntrypoint.nk-dist-1.17.1.jar:1.17.1]
> at org.apache.flink.runtime.entrypoint.ClusterEntrypoint:232)
> ~[flink-dist-1.17.1.jar:1.17.1]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8. 281]
> at javax.security.auth.Subject.doAs(Subject.java:422)~?:1.8.0 281]
> at org. apache.hadoop . security.UserGroupInformation. doAs
> (UserGroupInformation.
> java:1876)~[flink-shadd-hadoop-3-uber-3.1.1.7.2.1.0-327-9.0.jar:3.1.1.7.2.1.0-327-9.0]
> at org.apache.flink.runtime.security. contexts
> .HadoopSecurityContext.runSecured(HadoopSecurijava:41)
> ~[flink-dist-1.17.1.jar:1.17.1
> at org. apache.flink. runtime. entrypoint. ClusterEntrypoint .
> startCluster(clusterEntrypoint. java:229)link-dist-1.17.1.jar:1.17.1]...2
> more{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30649) Shutting down MiniCluster times out
[
https://issues.apache.org/jira/browse/FLINK-30649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767154#comment-17767154
]
Flaviu Cicio commented on FLINK-30649:
--
Hi, [~mapohl]
Can I have this one as my first ticket?
> Shutting down MiniCluster times out
> ---
>
> Key: FLINK-30649
> URL: https://issues.apache.org/jira/browse/FLINK-30649
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Test Infrastructure
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: stale-assigned, starter, test-stability
>
> {{Run kubernetes session test (default input)}} failed with a timeout.
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44748=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14=6317]
> It appears that there was some issue with shutting down the pods of the
> MiniCluster:
> {code:java}
> 2023-01-12T08:22:13.1388597Z timed out waiting for the condition on
> pods/flink-native-k8s-session-1-7dc9976688-gq788
> 2023-01-12T08:22:13.1390040Z timed out waiting for the condition on
> pods/flink-native-k8s-session-1-taskmanager-1-1 {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23446: [FLINK-32976][runtime] Fix NullPointException when starting flink cluster in standalone mode
flinkbot commented on PR #23446: URL: https://github.com/apache/flink/pull/23446#issuecomment-1728047395 ## CI report: * b75bf886971e4dc1ec4567177e2bb94cd28a3be4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler
[
https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767162#comment-17767162
]
Rui Fan commented on FLINK-33123:
-
Hi [~Zhanghao Chen] , thanks for your report!
Your analysis is correct, and I also found this bug last month. I didn't report
it because I don't know how to fix it properly.
First of all, I prefer changing it from FORWARD to REBALANCE, and I said the
reason at [code
review|https://github.com/apache/flink/pull/21443#discussion_r1042919428] of
FLINK-30213.
Why I don't know how to fix it properly?
As you said: we should changing the distribution type to ALL_TO_ALL in
jobgraph. Here is some issues here:
* issue1: The jobGraph is just generated once for Adaptive Scheduler. And the
jobgraph isn't changed even if the parallelism is changed(Just update the
ExecutionGraph).
* If the issue1 is solved, the issue2 is how to handle the case that from
REBALANCE to FORWARD?
** Assume a job has taskA and taskB, the parallelism of them are 3, and user
uses the FORWARD partitioner
** Time1: the parallelism of taskA is changed to 2, we should do 2 things:
*** replace FORWARD partitioner by REBALANCE partitioner in StreamTask
*** changing the distribution type to ALL_TO_ALL in jobgraph.
** Time2: the parallelism of taskB is changed to 2, we should do 2 things:
*** Using the FORWARD partitioner
*** Using the POINTWISE distribution type
** The case is fine.
** However, assume a job has taskA and taskB, the parallelism of them are 3,
and user uses the *REBALANCE* partitioner.
*** This case, user choose the REBALANCE partitioner even if it can use
FORWARD partitioner here.
*** For this case, we should still keep the REBALANCE partitioner and
ALL_TO_ALL for time1 and time2.
** So time2 needs to consider should we update it to FORWARD partitioner and
POINTWISE distribution type.
I'm not sure whether these 2 issues are clear.
Please let me know if I'm wrong, thanks~
> Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for
> autoscaler and adaptive scheduler
> ---
>
> Key: FLINK-33123
> URL: https://issues.apache.org/jira/browse/FLINK-33123
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler, Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Zhanghao Chen
>Priority: Critical
> Attachments: image-2023-09-20-15-09-22-733.png,
> image-2023-09-20-15-14-04-679.png
>
>
> *Background*
> https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
> wrong when the parallelism is changed for a vertex with a FORWARD edge, which
> is used by both the autoscaler and adaptive scheduler where one can change
> the vertex parallelism dynamically. Fix is applied to dynamically replace
> partitioner from FORWARD to REBLANCE on task deployment in
> {{{}StreamTask{}}}:
>
> !image-2023-09-20-15-09-22-733.png|width=560,height=221!
> *Problem*
> Unfortunately, the fix is still buggy in two aspects:
> # The connections between upstream and downstream tasks are determined by
> the distribution type of the partitioner when generating execution graph on
> the JM side. When the edge is FORWARD, the distribution type is POINTWISE,
> and Flink will try to evenly distribute subpartitions to all downstream
> tasks. If one want to change it to REBALANCE, the distribution type has to be
> changed to ALL_TO_ALL to make all-to-all connections between upstream and
> downstream tasks. However, the fix did not change the distribution type which
> makes the network connections be set up in a wrong way.
> # The FOWARD partitioner will be replaced if
> environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
> task parallelism. However, the number of subpartitions here equals to the
> number of downstream tasks of this particular task, which is also determined
> by the distribution type of the partitioner when generating execution graph
> on the JM side. When ceil(downstream task parallelism / upstream task
> parallelism) = upstream task parallelism, we will have the number of
> subpartitions = task parallelism. For example, for a topology A (parallelism
> 2) -> B (parallelism 5), we will have 1 A task having 2 subpartitions, 1 A
> task having 3 subpartition, and hence 1 task will have its number of
> subpartitions equals to the task parallelism 2 and skip partitioner
> replacement. As a result, that task will only send data to only one
> downstream task as the FORWARD partitioner always send data to the first
> subpartition. In fact, for a normal job with a FORWARD edge without any
> autoscaling action, you will find that the partitioner is changed to
> REBALANCE internally as the number of subpartitions always
[GitHub] [flink] hackergin opened a new pull request, #23446: [FLINK-32976][runtime] Fix NullPointException when starting flink cluster in standalone mode
hackergin opened a new pull request, #23446: URL: https://github.com/apache/flink/pull/23446 ## What is the purpose of the change *Fix NullPointException when starting flink cluster in standalone mode with hadoop configuration* ## Brief change log - Add default value when getting `DeploymentOptions.TARGET` options ## Verifying this change This change added tests and can be verified as follows: - In the HadoopFSDelegationTokenProviderITCase class, `obtainDelegationTokenWithStandaloneDeployment` has been added to test this modification. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and
[
https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhanghao Chen updated FLINK-33123:
--
Description:
*Background*
https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
wrong when the parallelism is changed for a vertex with a FORWARD edge, which
is used by both the autoscaler and adaptive scheduler where one can change the
vertex parallelism dynamically. Fix is applied to dynamically replace
partitioner from FORWARD to REBLANCE on task deployment in {{{}StreamTask{}}}:
!image-2023-09-20-15-09-22-733.png|width=560,height=221!
*Problem*
Unfortunately, the fix is still buggy in two aspects:
# The connections between upstream and downstream tasks are determined by the
distribution type of the partitioner when generating execution graph on the JM
side. When the edge is FORWARD, the distribution type is POINTWISE, and Flink
will try to evenly distribute subpartitions to all downstream tasks. If one
want to change it to REBALANCE, the distribution type has to be changed to
ALL_TO_ALL to make all-to-all connections between upstream and downstream
tasks. However, the fix did not change the distribution type which makes the
network connections be set up in a wrong way.
# The FOWARD partitioner will be replaced if
environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
task parallelism. However, the number of subpartitions here equals to the
number of downstream tasks of this particular task, which is also determined by
the distribution type of the partitioner when generating execution graph on the
JM side. When ceil(downstream task parallelism / upstream task parallelism) =
upstream task parallelism, we will have the number of subpartitions = task
parallelism. For example, for a topology A (parallelism 3) -> B (parallelism
8), we will have 2 A tasks having 3 subpartitions, 1 A task having 2
subpartition, and hence 2 tasks will have its number of subpartitions equals to
the task parallelism 3 and skip partitioner replacement. In fact, for a normal
job with a FORWARD edge without any autoscaling action, you will find that the
partitioner is changed to REBALANCE internally as the number of subpartitions
always equals to 1 in this case.
was:
*Background*
https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
wrong when the parallelism is changed for a vertex with a FORWARD edge, which
is used by both the autoscaler and adaptive scheduler where one can change the
vertex parallelism dynamically. Fix is applied to dynamically replace
partitioner from FORWARD to REBLANCE on task deployment in {{{}StreamTask{}}}:
{{private static void
replaceForwardPartitionerIfConsumerParallelismDoesNotMatch(}}
{{ Environment environment, NonChainedOutput streamOutput) {}}
{{ Environment environment, NonChainedOutput streamOutput, int
outputIndex) {}}
{{ if (streamOutput.getPartitioner() instanceof ForwardPartitioner}}
{{ && streamOutput.getConsumerParallelism()}}
{{ &&
environment.getWriter(outputIndex).getNumberOfSubpartitions()}}
{{ !=
environment.getTaskInfo().getNumberOfParallelSubtasks()) {}}
{{ LOG.debug(}}
{{ "Replacing forward partitioner with rebalance for {}",}}
{{ environment.getTaskInfo().getTaskNameWithSubtasks());}}
{{ streamOutput.setPartitioner(new RebalancePartitioner<>());}}
{{ }}}
{{ }}}
*Problem*
Unfortunately, the fix is still buggy in two aspects:
# The connections between upstream and downstream tasks are determined by the
distribution type of the partitioner when generating execution graph on the JM
side. When the edge is FORWARD, the distribution type is POINTWISE, and Flink
will try to evenly distribute subpartitions to all downstream tasks. If one
want to change it to REBALANCE, the distribution type has to be changed to
ALL_TO_ALL to make all-to-all connections between upstream and downstream
tasks. However, the fix did not change the distribution type which makes the
network connections be set up in a wrong way.
# The FOWARD partitioner will be replaced if
environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
task parallelism. However, the number of subpartitions here equals to the
number of downstream tasks of this particular task, which is also determined by
the distribution type of the partitioner when generating execution graph on the
JM side. When ceil(downstream task parallelism / upstream task parallelism) =
upstream task parallelism, we will have the number of subpartitions = task
parallelism. In fact, for a normal job with a FORWARD edge without any
autoscaling action, you will find that the partitioner is changed to REBALANCE
internally as the number of subpartitions always equals
[jira] [Commented] (FLINK-33052) codespeed and benchmark server is down
[
https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767153#comment-17767153
]
Piotr Nowojski commented on FLINK-33052:
I think there is no big difference one way or the other. The {{dak8s.net}} is
owned by Ververica, so [~jingge] should be able to redirect the old one. If
it's easier to buy new one, let's do that.
> codespeed and benchmark server is down
> --
>
> Key: FLINK-33052
> URL: https://issues.apache.org/jira/browse/FLINK-33052
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks, Test Infrastructure
>Affects Versions: 1.18.0
>Reporter: Jing Ge
>Assignee: Zakelly Lan
>Priority: Blocker
>
> No update in #flink-dev-benchmarks slack channel since 25th August.
> It was a EC2 running in a legacy aws account. Currently on one knows which
> account it is.
>
> https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33121) Failed precondition in JobExceptionsHandler due to concurrent global failures
[
https://issues.apache.org/jira/browse/FLINK-33121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767029#comment-17767029
]
Chesnay Schepler commented on FLINK-33121:
--
??there are rare cases where multiple concurrent global failures are triggered??
Can you give us an example of when this might happen?
> Failed precondition in JobExceptionsHandler due to concurrent global failures
> -
>
> Key: FLINK-33121
> URL: https://issues.apache.org/jira/browse/FLINK-33121
> Project: Flink
> Issue Type: Bug
>Reporter: Panagiotis Garefalakis
>Priority: Major
>
> {{JobExceptionsHandler#createRootExceptionInfo}} *only* allows concurrent
> exceptions that are local failures *--* otherwise throws an assertion as part
> of {{{}asserLocalExceptionInfo{}}}.
> However, there are rare cases where multiple concurrent global failures are
> triggered and added to the failureCollection, before transitioning the job
> state to Failed e.g., through {{StateWithExecutionGraph#handleGlobalFailure}}
> of the AdaptiveScheduler.
> In this case the last added will be the root and the next one will trigger
> the assertion
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and
[
https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhanghao Chen updated FLINK-33123:
--
Attachment: image-2023-09-20-15-09-22-733.png
> Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for
> autoscaler and adaptive scheduler and
> -
>
> Key: FLINK-33123
> URL: https://issues.apache.org/jira/browse/FLINK-33123
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler, Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Zhanghao Chen
>Priority: Critical
> Attachments: image-2023-09-20-15-09-22-733.png
>
>
> *Background*
> https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
> wrong when the parallelism is changed for a vertex with a FORWARD edge, which
> is used by both the autoscaler and adaptive scheduler where one can change
> the vertex parallelism dynamically. Fix is applied to dynamically replace
> partitioner from FORWARD to REBLANCE on task deployment in
> {{{}StreamTask{}}}:
> {{private static void
> replaceForwardPartitionerIfConsumerParallelismDoesNotMatch(}}
> {{ Environment environment, NonChainedOutput streamOutput) {}}
> {{ Environment environment, NonChainedOutput streamOutput, int
> outputIndex) {}}
> {{ if (streamOutput.getPartitioner() instanceof ForwardPartitioner}}
> {{ && streamOutput.getConsumerParallelism()}}
> {{ &&
> environment.getWriter(outputIndex).getNumberOfSubpartitions()}}
> {{ !=
> environment.getTaskInfo().getNumberOfParallelSubtasks()) {}}
> {{ LOG.debug(}}
> {{ "Replacing forward partitioner with rebalance for {}",}}
> {{ environment.getTaskInfo().getTaskNameWithSubtasks());}}
> {{ streamOutput.setPartitioner(new RebalancePartitioner<>());}}
> {{ }}}
> {{ }}}
> *Problem*
> Unfortunately, the fix is still buggy in two aspects:
> # The connections between upstream and downstream tasks are determined by
> the distribution type of the partitioner when generating execution graph on
> the JM side. When the edge is FORWARD, the distribution type is POINTWISE,
> and Flink will try to evenly distribute subpartitions to all downstream
> tasks. If one want to change it to REBALANCE, the distribution type has to be
> changed to ALL_TO_ALL to make all-to-all connections between upstream and
> downstream tasks. However, the fix did not change the distribution type which
> makes the network connections be set up in a wrong way.
> # The FOWARD partitioner will be replaced if
> environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
> task parallelism. However, the number of subpartitions here equals to the
> number of downstream tasks of this particular task, which is also determined
> by the distribution type of the partitioner when generating execution graph
> on the JM side. When ceil(downstream task parallelism / upstream task
> parallelism) = upstream task parallelism, we will have the number of
> subpartitions = task parallelism. In fact, for a normal job with a FORWARD
> edge without any autoscaling action, you will find that the partitioner is
> changed to REBALANCE internally as the number of subpartitions always equals
> to 1 in this case.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and
Zhanghao Chen created FLINK-33123:
-
Summary: Wrong dynamic replacement of partitioner from FORWARD to
REBLANCE for autoscaler and adaptive scheduler and
Key: FLINK-33123
URL: https://issues.apache.org/jira/browse/FLINK-33123
Project: Flink
Issue Type: Bug
Components: Autoscaler, Runtime / Coordination
Affects Versions: 1.17.0, 1.18.0
Reporter: Zhanghao Chen
*Background*
https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
wrong when the parallelism is changed for a vertex with a FORWARD edge, which
is used by both the autoscaler and adaptive scheduler where one can change the
vertex parallelism dynamically. Fix is applied to dynamically replace
partitioner from FORWARD to REBLANCE on task deployment in {{{}StreamTask{}}}:
{{private static void
replaceForwardPartitionerIfConsumerParallelismDoesNotMatch(}}
{{ Environment environment, NonChainedOutput streamOutput) {}}
{{ Environment environment, NonChainedOutput streamOutput, int
outputIndex) {}}
{{ if (streamOutput.getPartitioner() instanceof ForwardPartitioner}}
{{ && streamOutput.getConsumerParallelism()}}
{{ &&
environment.getWriter(outputIndex).getNumberOfSubpartitions()}}
{{ !=
environment.getTaskInfo().getNumberOfParallelSubtasks()) {}}
{{ LOG.debug(}}
{{ "Replacing forward partitioner with rebalance for {}",}}
{{ environment.getTaskInfo().getTaskNameWithSubtasks());}}
{{ streamOutput.setPartitioner(new RebalancePartitioner<>());}}
{{ }}}
{{ }}}
*Problem*
Unfortunately, the fix is still buggy in two aspects:
# The connections between upstream and downstream tasks are determined by the
distribution type of the partitioner when generating execution graph on the JM
side. When the edge is FORWARD, the distribution type is POINTWISE, and Flink
will try to evenly distribute subpartitions to all downstream tasks. If one
want to change it to REBALANCE, the distribution type has to be changed to
ALL_TO_ALL to make all-to-all connections between upstream and downstream
tasks. However, the fix did not change the distribution type which makes the
network connections be set up in a wrong way.
# The FOWARD partitioner will be replaced if
environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
task parallelism. However, the number of subpartitions here equals to the
number of downstream tasks of this particular task, which is also determined by
the distribution type of the partitioner when generating execution graph on the
JM side. When ceil(downstream task parallelism / upstream task parallelism) =
upstream task parallelism, we will have the number of subpartitions = task
parallelism. In fact, for a normal job with a FORWARD edge without any
autoscaling action, you will find that the partitioner is changed to REBALANCE
internally as the number of subpartitions always equals to 1 in this case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler
[ https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767162#comment-17767162 ] Rui Fan edited comment on FLINK-33123 at 9/20/23 3:30 PM: -- Hi [~Zhanghao Chen] , thanks for your report! Your analysis is correct, and I also found this bug last month. I didn't report it because I don't know how to fix it properly and the FORWARD partitioner between 2 tasks aren't common(most of operator are chained directly). First of all, I prefer changing it from FORWARD to REBALANCE, and I said the reason at [code review|https://github.com/apache/flink/pull/21443#discussion_r1042919428] of FLINK-30213. Why I don't know how to fix it properly? As you said: we should changing the distribution type to ALL_TO_ALL in jobgraph. Here is some issues here: * issue1: The jobGraph is just generated once for Adaptive Scheduler. And the jobgraph isn't changed even if the parallelism is changed(Just update the ExecutionGraph). * If the issue1 is solved, the issue2 is how to handle the case that from REBALANCE to FORWARD? ** Assume a job has taskA and taskB, the parallelism of them are 3, and user uses the FORWARD partitioner ** Time1: the parallelism of taskA is changed to 2, we should do 2 things: *** replace FORWARD partitioner by REBALANCE partitioner in StreamTask *** changing the distribution type to ALL_TO_ALL in jobgraph. ** Time2: the parallelism of taskB is changed to 2, we should do 2 things: *** Using the FORWARD partitioner *** Using the POINTWISE distribution type ** The case is fine. ** However, assume a job has taskA and taskB, the parallelism of them are 3, and user uses the *REBALANCE* partitioner. *** This case, user choose the REBALANCE partitioner even if it can use FORWARD partitioner here. *** For this case, we should still keep the REBALANCE partitioner and ALL_TO_ALL for time1 and time2. ** So time2 needs to consider should we update it to FORWARD partitioner and POINTWISE distribution type. I'm not sure whether these 2 issues are clear. Please let me know if I'm wrong, thanks~ was (Author: fanrui): Hi [~Zhanghao Chen] , thanks for your report! Your analysis is correct, and I also found this bug last month. I didn't report it because I don't know how to fix it properly. First of all, I prefer changing it from FORWARD to REBALANCE, and I said the reason at [code review|https://github.com/apache/flink/pull/21443#discussion_r1042919428] of FLINK-30213. Why I don't know how to fix it properly? As you said: we should changing the distribution type to ALL_TO_ALL in jobgraph. Here is some issues here: * issue1: The jobGraph is just generated once for Adaptive Scheduler. And the jobgraph isn't changed even if the parallelism is changed(Just update the ExecutionGraph). * If the issue1 is solved, the issue2 is how to handle the case that from REBALANCE to FORWARD? ** Assume a job has taskA and taskB, the parallelism of them are 3, and user uses the FORWARD partitioner ** Time1: the parallelism of taskA is changed to 2, we should do 2 things: *** replace FORWARD partitioner by REBALANCE partitioner in StreamTask *** changing the distribution type to ALL_TO_ALL in jobgraph. ** Time2: the parallelism of taskB is changed to 2, we should do 2 things: *** Using the FORWARD partitioner *** Using the POINTWISE distribution type ** The case is fine. ** However, assume a job has taskA and taskB, the parallelism of them are 3, and user uses the *REBALANCE* partitioner. *** This case, user choose the REBALANCE partitioner even if it can use FORWARD partitioner here. *** For this case, we should still keep the REBALANCE partitioner and ALL_TO_ALL for time1 and time2. ** So time2 needs to consider should we update it to FORWARD partitioner and POINTWISE distribution type. I'm not sure whether these 2 issues are clear. Please let me know if I'm wrong, thanks~ > Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for > autoscaler and adaptive scheduler > --- > > Key: FLINK-33123 > URL: https://issues.apache.org/jira/browse/FLINK-33123 > Project: Flink > Issue Type: Bug > Components: Autoscaler, Runtime / Coordination >Affects Versions: 1.17.0, 1.18.0 >Reporter: Zhanghao Chen >Priority: Critical > Attachments: image-2023-09-20-15-09-22-733.png, > image-2023-09-20-15-14-04-679.png > > > *Background* > https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is > wrong when the parallelism is changed for a vertex with a FORWARD edge, which > is used by both the autoscaler and adaptive scheduler where one can change > the vertex parallelism dynamically. Fix is applied to dynamically replace > partitioner
[GitHub] [flink] Aoboyan closed pull request #20926: Update ch temporal_table_function.md
Aoboyan closed pull request #20926: Update ch temporal_table_function.md URL: https://github.com/apache/flink/pull/20926 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33111) Flink Pulsar Connector to Pulsar Client Version Mismatch
[ https://issues.apache.org/jira/browse/FLINK-33111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767131#comment-17767131 ] Zili Chen commented on FLINK-33111: --- We upgrade the Pulsar client version in https://github.com/apache/flink-connector-pulsar/pull/25 which brings benefits for the new version. Perhaps we should update the document and people who use Pulsar 2.10.x can use 3.x connector. > Flink Pulsar Connector to Pulsar Client Version Mismatch > > > Key: FLINK-33111 > URL: https://issues.apache.org/jira/browse/FLINK-33111 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.17.1 >Reporter: Jason Kania >Priority: Major > > In the documentation for the Flink Pulsar Connector, > ([https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/pulsar/]) > it indicates that 2.10.0 and above versions of the pulsar client are > supported "You can use the connector with the Pulsar 2.10.0 or higher" and > the pom file entry references the 4.0.0-1.17 version of the connector which > points to the 2.11.0 version of the Pulsar client. However, when using Pulsar > Client 2.10.4 or 2.10.5, the following error is generated: > > java.lang.NoSuchMethodError: 'org.apache.pulsar.client.api.ClientBuilder > org.apache.pulsar.client.api.ClientBuilder.connectionMaxIdleSeconds(int)' > at > org.apache.flink.connector.pulsar.common.config.PulsarClientFactory.createClient(PulsarClientFactory.java:127) > at > org.apache.flink.connector.pulsar.source.reader.PulsarSourceReader.create(PulsarSourceReader.java:266) > at > org.apache.flink.connector.pulsar.source.PulsarSource.createReader(PulsarSource.java:137) > at > org.apache.flink.streaming.api.operators.SourceOperator.initReader(SourceOperator.java:312) > at > org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask.init(SourceOperatorStreamTask.java:93) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:699) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675) > at > org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) > at > org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) > at java.base/java.lang.Thread.run(Thread.java:829) > > The referenced method 'connectionMaxIdleSeconds' is only available in the > Pulsar 2.11 client when looking at the source code. I am not sure whether the > documentation is wrong and the Flink Pulsar Connector 2.11 is the intended > Pulsar version. However, my understanding is that Pulsar 2.11 is targeted > toward java 17. This would create the need for mixed Java 11 and Java 17 > deployment unless the Pulsar client code is compiled for 2.11. > > Documentation cleanup and a reference to the appropriate Java versions is > needed. A fix to the 1.17.1 Flink pulsar connector may alternatively be > required. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Aoboyan closed pull request #20923: Update temporal_table_function.md
Aoboyan closed pull request #20923: Update temporal_table_function.md URL: https://github.com/apache/flink/pull/20923 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] CuiYanxiang commented on pull request #23439: [FLINK-33117][table][docs] Fix scala example in udfs page
CuiYanxiang commented on PR #23439: URL: https://github.com/apache/flink/pull/23439#issuecomment-1727100811 Hi @wuchong , would you mind helping review this pr? thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] huwh merged pull request #23243: [FLINK-32846][runtime][JUnit5 Migration] The metrics package of flink-runtime module
huwh merged PR #23243: URL: https://github.com/apache/flink/pull/23243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and
[
https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhanghao Chen updated FLINK-33123:
--
Description:
*Background*
https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
wrong when the parallelism is changed for a vertex with a FORWARD edge, which
is used by both the autoscaler and adaptive scheduler where one can change the
vertex parallelism dynamically. Fix is applied to dynamically replace
partitioner from FORWARD to REBLANCE on task deployment in {{{}StreamTask{}}}:
!image-2023-09-20-15-09-22-733.png|width=560,height=221!
*Problem*
Unfortunately, the fix is still buggy in two aspects:
# The connections between upstream and downstream tasks are determined by the
distribution type of the partitioner when generating execution graph on the JM
side. When the edge is FORWARD, the distribution type is POINTWISE, and Flink
will try to evenly distribute subpartitions to all downstream tasks. If one
want to change it to REBALANCE, the distribution type has to be changed to
ALL_TO_ALL to make all-to-all connections between upstream and downstream
tasks. However, the fix did not change the distribution type which makes the
network connections be set up in a wrong way.
# The FOWARD partitioner will be replaced if
environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
task parallelism. However, the number of subpartitions here equals to the
number of downstream tasks of this particular task, which is also determined by
the distribution type of the partitioner when generating execution graph on the
JM side. When ceil(downstream task parallelism / upstream task parallelism) =
upstream task parallelism, we will have the number of subpartitions = task
parallelism. For example, for a topology A (parallelism 2) -> B (parallelism
5), we will have 1 A task having 2 subpartitions, 1 A task having 3
subpartition, and hence 1 task will have its number of subpartitions equals to
the task parallelism 2 and skip partitioner replacement. As a result, that task
will only send data to only one downstream task as the FORWARD partitioner
always send data to the first subpartition. In fact, for a normal job with a
FORWARD edge without any autoscaling action, you will find that the partitioner
is changed to REBALANCE internally as the number of subpartitions always equals
to 1 in this case.
!image-2023-09-20-15-14-04-679.png|width=892,height=301!
was:
*Background*
https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
wrong when the parallelism is changed for a vertex with a FORWARD edge, which
is used by both the autoscaler and adaptive scheduler where one can change the
vertex parallelism dynamically. Fix is applied to dynamically replace
partitioner from FORWARD to REBLANCE on task deployment in {{{}StreamTask{}}}:
!image-2023-09-20-15-09-22-733.png|width=560,height=221!
*Problem*
Unfortunately, the fix is still buggy in two aspects:
# The connections between upstream and downstream tasks are determined by the
distribution type of the partitioner when generating execution graph on the JM
side. When the edge is FORWARD, the distribution type is POINTWISE, and Flink
will try to evenly distribute subpartitions to all downstream tasks. If one
want to change it to REBALANCE, the distribution type has to be changed to
ALL_TO_ALL to make all-to-all connections between upstream and downstream
tasks. However, the fix did not change the distribution type which makes the
network connections be set up in a wrong way.
# The FOWARD partitioner will be replaced if
environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
task parallelism. However, the number of subpartitions here equals to the
number of downstream tasks of this particular task, which is also determined by
the distribution type of the partitioner when generating execution graph on the
JM side. When ceil(downstream task parallelism / upstream task parallelism) =
upstream task parallelism, we will have the number of subpartitions = task
parallelism. For example, for a topology A (parallelism 3) -> B (parallelism
8), we will have 2 A tasks having 3 subpartitions, 1 A task having 2
subpartition, and hence 2 tasks will have its number of subpartitions equals to
the task parallelism 3 and skip partitioner replacement. In fact, for a normal
job with a FORWARD edge without any autoscaling action, you will find that the
partitioner is changed to REBALANCE internally as the number of subpartitions
always equals to 1 in this case.
> Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for
> autoscaler and adaptive scheduler and
> -
>
> Key: FLINK-33123
> URL:
[jira] [Updated] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and
[
https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhanghao Chen updated FLINK-33123:
--
Attachment: image-2023-09-20-15-14-04-679.png
> Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for
> autoscaler and adaptive scheduler and
> -
>
> Key: FLINK-33123
> URL: https://issues.apache.org/jira/browse/FLINK-33123
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler, Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Zhanghao Chen
>Priority: Critical
> Attachments: image-2023-09-20-15-09-22-733.png,
> image-2023-09-20-15-14-04-679.png
>
>
> *Background*
> https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
> wrong when the parallelism is changed for a vertex with a FORWARD edge, which
> is used by both the autoscaler and adaptive scheduler where one can change
> the vertex parallelism dynamically. Fix is applied to dynamically replace
> partitioner from FORWARD to REBLANCE on task deployment in
> {{{}StreamTask{}}}:
>
> !image-2023-09-20-15-09-22-733.png|width=560,height=221!
> *Problem*
> Unfortunately, the fix is still buggy in two aspects:
> # The connections between upstream and downstream tasks are determined by
> the distribution type of the partitioner when generating execution graph on
> the JM side. When the edge is FORWARD, the distribution type is POINTWISE,
> and Flink will try to evenly distribute subpartitions to all downstream
> tasks. If one want to change it to REBALANCE, the distribution type has to be
> changed to ALL_TO_ALL to make all-to-all connections between upstream and
> downstream tasks. However, the fix did not change the distribution type which
> makes the network connections be set up in a wrong way.
> # The FOWARD partitioner will be replaced if
> environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
> task parallelism. However, the number of subpartitions here equals to the
> number of downstream tasks of this particular task, which is also determined
> by the distribution type of the partitioner when generating execution graph
> on the JM side. When ceil(downstream task parallelism / upstream task
> parallelism) = upstream task parallelism, we will have the number of
> subpartitions = task parallelism. For example, for a topology A (parallelism
> 3) -> B (parallelism 8), we will have 2 A tasks having 3 subpartitions, 1 A
> task having 2 subpartition, and hence 2 tasks will have its number of
> subpartitions equals to the task parallelism 3 and skip partitioner
> replacement. In fact, for a normal job with a FORWARD edge without any
> autoscaling action, you will find that the partitioner is changed to
> REBALANCE internally as the number of subpartitions always equals to 1 in
> this case.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol commented on a diff in pull request #23445: [FLINK-15736][docs] Add Java compatibility page
zentol commented on code in PR #23445: URL: https://github.com/apache/flink/pull/23445#discussion_r1331743142 ## docs/content.zh/docs/deployment/java_compatibility.md: ## @@ -0,0 +1,77 @@ +--- +title: Java Compatibility +weight: 2 +type: docs +--- + + +# Java compatibility + +This page lists which Java versions Flink supports and what limitations apply (if any). + +## Java 8 (deprecated) + +Support for Java 8 has been deprecated in 1.15.0. +It is recommended to migrate to Java 11. + +## Java 11 + +Support for Java 17 was added in 1.10.0 and is the recommended Java version to run Flink on. Review Comment: yes :see_no_evil: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767128#comment-17767128
]
Rui Fan commented on FLINK-33109:
-
Could you try this image? It's the offcial image based on release-1.17 branch.
[https://github.com/apache/flink-docker/pkgs/container/flink-docker/128728529?tag=1.17-SNAPSHOT-scala_2.12-java11-debian]
Also, would you mind sharing a flink job demo that can reproduce your bug? If
so, I can try it with the latest code.
> Watermark alignment not applied after recovery from checkpoint
> --
>
> Key: FLINK-33109
> URL: https://issues.apache.org/jira/browse/FLINK-33109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.1
>Reporter: Yordan Pavlov
>Priority: Major
> Attachments: image-2023-09-18-15-40-06-868.png,
> image-2023-09-18-15-46-16-106.png
>
>
> I am observing a problem where after recovery from a checkpoint the Kafka
> source watermarks would start to diverge not honoring the watermark alignment
> setting I have applied.
> I have a Kafka source which reads a topic with 32 partitions. I am applying
> the following watermark strategy:
> {code:java}
> new EventAwareWatermarkStrategy[KeyedKafkaSourceMessage]](msg =>
> msg.value.getTimestamp)
> .withWatermarkAlignment("alignment-sources-group",
> time.Duration.ofMillis(sourceWatermarkAlignmentBlocks)){code}
>
> This works great up until my job needs to recover from checkpoint. Once the
> recovery takes place, no alignment is taking place any more. This can best be
> illustrated by looking at the watermark metrics for various operators in the
> image:
> !image-2023-09-18-15-40-06-868.png!
>
> You can see how the watermarks disperse after the recovery. Trying to debug
> the problem I noticed that before the failure there would be calls in
>
> {code:java}
> SourceCoordinator::announceCombinedWatermark()
> {code}
> after the recovery, no calls get there, so no value for
> {code:java}
> watermarkAlignmentParams.getMaxAllowedWatermarkDrift(){code}
> is ever read. I can manually fix the problem If I stop the job, clear all
> state from Zookeeper and then manually start Flink providing the last
> checkpoint with
> {code:java}
> '–fromSavepoint'{code}
> flag. This would cause the SourceCoordinator to be constructed properly and
> watermark drift to be checked. Once recovery manually watermarks would again
> converge to the allowed drift as seen in the metrics:
> !image-2023-09-18-15-46-16-106.png!
>
> Let me know If I can be helpful by providing any more information.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33122) [Benchmark] Null checkpoint directory in rescaling benchmarks
Zakelly Lan created FLINK-33122: --- Summary: [Benchmark] Null checkpoint directory in rescaling benchmarks Key: FLINK-33122 URL: https://issues.apache.org/jira/browse/FLINK-33122 Project: Flink Issue Type: Bug Components: Benchmarks Reporter: Zakelly Lan Assignee: Zakelly Lan Currently, when setting up a rescaling benchmark, a local checkpoint storage is created based on a local path configured by "benchmark.state.data-dir". When user does not provide value for this option, an exception is thrown. In this case, the right behavior should be to create a temporary directory for checkpoint, just like the _StateBackendBenchmarkUtils#createKeyedStateBackend_ does for local data directory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] snuyanzin commented on a diff in pull request #23445: [FLINK-15736][docs] Add Java compatibility page
snuyanzin commented on code in PR #23445: URL: https://github.com/apache/flink/pull/23445#discussion_r1331730072 ## docs/content.zh/docs/deployment/java_compatibility.md: ## @@ -0,0 +1,77 @@ +--- +title: Java Compatibility +weight: 2 +type: docs +--- + + +# Java compatibility + +This page lists which Java versions Flink supports and what limitations apply (if any). + +## Java 8 (deprecated) + +Support for Java 8 has been deprecated in 1.15.0. +It is recommended to migrate to Java 11. + +## Java 11 + +Support for Java 17 was added in 1.10.0 and is the recommended Java version to run Flink on. Review Comment: ```suggestion Support for Java 11 was added in 1.10.0 and is the recommended Java version to run Flink on. ``` should it be 11 here? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] huwh commented on a diff in pull request #23242: [FLINK-32851][runtime][JUnit5 Migration] The rest package of flink-runtime module
huwh commented on code in PR #23242:
URL: https://github.com/apache/flink/pull/23242#discussion_r1328092633
##
flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/async/CompletedOperationCacheTest.java:
##
@@ -154,35 +149,35 @@ public void testCacheTimeoutCanBeConfigured() throws
Exception {
manualTicker.advanceTime(baseTimeout.multipliedBy(2).getSeconds(),
TimeUnit.SECONDS);
-
assertTrue(completedOperationCache.get(TEST_OPERATION_KEY).isPresent());
+
assertThat(completedOperationCache.get(TEST_OPERATION_KEY)).isPresent();
}
@Test
-public void containsReturnsFalseForUnknownOperation() {
-
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY),
is(false));
+void containsReturnsFalseForUnknownOperation() {
+
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY)).isFalse();
}
@Test
-public void containsChecksOnoingOperations() {
+void containsChecksOnoingOperations() {
completedOperationCache.registerOngoingOperation(
TEST_OPERATION_KEY, new CompletableFuture<>());
-
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY),
is(true));
+
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY)).isTrue();
}
@Test
-public void containsChecksCompletedOperations() {
+void containsChecksCompletedOperations() {
completedOperationCache.registerOngoingOperation(
TEST_OPERATION_KEY, CompletableFuture.completedFuture(null));
-
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY),
is(true));
+
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY)).isTrue();
}
@Test
-public void containsDoesNotMarkResultAsAccessed() {
+void containsDoesNotMarkResultAsAccessed() {
completedOperationCache.registerOngoingOperation(
TEST_OPERATION_KEY, CompletableFuture.completedFuture(null));
-
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY),
is(true));
-assertThat(
-completedOperationCache.closeAsync(),
-FlinkMatchers.willNotComplete(Duration.ofMillis(10)));
+
assertThat(completedOperationCache.containsOperation(TEST_OPERATION_KEY)).isTrue();
+
+assertThat(completedOperationCache.closeAsync())
Review Comment:
```suggestion
assertThatFuture(completedOperationCache.closeAsync())
.willNotCompleteWithin(Duration.ofMillis(10));
```
##
flink-runtime/src/test/java/org/apache/flink/runtime/rest/RestServerEndpointITCase.java:
##
@@ -543,21 +529,23 @@ public void testDefaultVersionRouting() throws Exception {
.build();
try (final Response response = client.newCall(request).execute()) {
-assertEquals(HttpResponseStatus.ACCEPTED.code(), response.code());
+
assertThat(response.code()).isEqualTo(HttpResponseStatus.ACCEPTED.code());
}
}
-@Test
-public void testNonSslRedirectForEnabledSsl() throws Exception {
-Assume.assumeTrue(config.getBoolean(SecurityOptions.SSL_REST_ENABLED));
+@TestTemplate
+void testNonSslRedirectForEnabledSsl() throws Exception {
+if (!config.getBoolean(SecurityOptions.SSL_REST_ENABLED)) {
Review Comment:
ditto
##
flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/metrics/AggregatingMetricsHandlerTestBase.java:
##
@@ -185,9 +183,9 @@ public void getStores() throws Exception {
.sorted()
.collect(Collectors.toList());
-assertEquals(1, sortedMetrics1.size());
+assertThat(sortedMetrics1).hasSize(1);
Review Comment:
ditto
##
flink-runtime/src/test/java/org/apache/flink/runtime/rest/messages/json/JobResultDeserializerTest.java:
##
@@ -78,20 +74,19 @@ public void testInvalidType() throws Exception {
+ "}",
JobResult.class);
} catch (final JsonMappingException e) {
-assertThat(
-e.getMessage(),
-containsString("Expected token VALUE_NUMBER_INT (was
VALUE_STRING)"));
+assertThat(e.getMessage())
Review Comment:
```suggestion
assertThatThrownBy(
() ->
objectMapper.readValue(
"{\n"
+ "\t\"id\":
\"1bb5e8c7df49938733b7c6a73678de6a\",\n"
+ "\t\"net-runtime\":
\"invalid\"\n"
+ "}",
JobResult.class))
[GitHub] [flink-benchmarks] FangYongs commented on a diff in pull request #78: [FLINK-33033][olap][haservice] Add haservice micro benchmark for olap
FangYongs commented on code in PR #78:
URL: https://github.com/apache/flink-benchmarks/pull/78#discussion_r1331196199
##
src/main/java/org/apache/flink/olap/benchmark/HighAvailabilityServiceBenchmark.java:
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.olap.benchmark;
+
+import org.apache.curator.test.TestingServer;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.benchmark.BenchmarkBase;
+import org.apache.flink.benchmark.FlinkEnvironmentContext;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.HighAvailabilityOptions;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobmanager.HighAvailabilityMode;
+import org.apache.flink.runtime.testtasks.NoOpInvokable;
+import org.apache.flink.util.FileUtils;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+import org.openjdk.jmh.runner.options.VerboseMode;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.util.UUID;
+
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.openjdk.jmh.annotations.Scope.Thread;
+
+/** The benchmark for submitting short-lived jobs with and without high
availability service. */
+@OutputTimeUnit(SECONDS)
+public class HighAvailabilityServiceBenchmark extends BenchmarkBase {
+ public static void main(String[] args) throws RunnerException {
+ Options options =
+ new OptionsBuilder()
+ .verbosity(VerboseMode.NORMAL)
+ .include(".*" +
HighAvailabilityServiceBenchmark.class.getCanonicalName() + ".*")
+ .build();
+
+ new Runner(options).run();
+ }
+
+ @Benchmark
+ public void submitJobThroughput(HighAvailabilityContext context) throws
Exception {
+ context.miniCluster.executeJobBlocking(buildNoOpJob());
+ }
+
+ private JobGraph buildNoOpJob() {
+ JobGraph jobGraph = new JobGraph(JobID.generate(),
UUID.randomUUID().toString());
+ jobGraph.addVertex(createNoOpVertex());
+ return jobGraph;
+ }
+
+ private JobVertex createNoOpVertex() {
+ JobVertex vertex = new JobVertex("v");
+ vertex.setInvokableClass(NoOpInvokable.class);
+ vertex.setParallelism(1);
+ vertex.setMaxParallelism(1);
+ return vertex;
+ }
+
+ @State(Thread)
+ public static class HighAvailabilityContext extends
FlinkEnvironmentContext {
+ private TestingServer testingServer;
+ public final File haDir;
+
+ @Param({"ZOOKEEPER", "NONE"})
+ public HighAvailabilityMode highAvailabilityMode;
+
+ public HighAvailabilityContext() {
+ try {
+ haDir =
Files.createTempDirectory("bench-ha-").toFile();
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ @Override
+ public void setUp() throws Exception {
+ if (isZookeeperHighAvailability()) {
+ testingServer = new TestingServer();
+ testingServer.start();
+ }
+
+ super.setUp();
Review Comment:
The method `super.setUp()` will call `createConfiguration()` to get
`Configuration` and create a `MiniCluster`. We need to start `TestingServer`
before `createConfiguration()`, then we can add zookeeper quorum in the
[jira] [Updated] (FLINK-33124) Kafka Connector not working for table
[ https://issues.apache.org/jira/browse/FLINK-33124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aarsh Shah updated FLINK-33124: --- Component/s: Connectors / Kafka Table SQL / Runtime Affects Version/s: 1.16.0 Description: The kafka table connector is not working for table [As mentioned here|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/#security], I am providing the mechanism, the protocol and the jaas config, but the server doesn't have a truststore, so is it mandatory to provide the truststore? And if not, what am I missing This is my command CREATE TABLE IF NOT EXISTS `xyz` ( proctime AS PROCTIME() ) WITH ( 'connector' = 'kafka', 'topic' = 'xyz', 'scan.startup.mode' = 'group-offsets', 'properties.auto.offset.reset' = 'earliest', 'properties.security.protocol' = 'SASL_SSL', 'properties.sasl.mechanism' = 'SCRAM-SHA-256', 'properties.sasl.jaas.config' = 'org.apache.kafka.common.security.scram.ScramLoginModule required username="xyz" password="xyz";', 'properties.bootstrap.servers' = 'xyz', 'properties.group.id' = 'xyz', 'format' = 'protobuf', 'protobuf.message-class-name' = 'xyz' > Kafka Connector not working for table > - > > Key: FLINK-33124 > URL: https://issues.apache.org/jira/browse/FLINK-33124 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Table SQL / Runtime >Affects Versions: 1.16.0 >Reporter: Aarsh Shah >Priority: Major > > The kafka table connector is not working for table > [As mentioned > here|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/table/kafka/#security], > I am providing the mechanism, the protocol and the jaas config, but the > server doesn't have a truststore, so is it mandatory to provide the > truststore? > And if not, what am I missing > This is my command > CREATE TABLE IF NOT EXISTS `xyz` ( > proctime AS PROCTIME() > ) WITH ( > 'connector' = 'kafka', > 'topic' = 'xyz', > 'scan.startup.mode' = 'group-offsets', > 'properties.auto.offset.reset' = 'earliest', > 'properties.security.protocol' = 'SASL_SSL', > 'properties.sasl.mechanism' = 'SCRAM-SHA-256', > 'properties.sasl.jaas.config' = > 'org.apache.kafka.common.security.scram.ScramLoginModule required > username="xyz" password="xyz";', > 'properties.bootstrap.servers' = 'xyz', > 'properties.group.id' = 'xyz', > 'format' = 'protobuf', > 'protobuf.message-class-name' = 'xyz' -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] ashangit opened a new pull request, #680: [FLINK-33125] Bump JOSDK to 4.4.4
ashangit opened a new pull request, #680: URL: https://github.com/apache/flink-kubernetes-operator/pull/680 ## What is the purpose of the change Upgrade JOSDK to 4.4.4 which contains fix on leader election stop not called ## Brief change log - Update dependencies ## Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): yes - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33059) Support transparent compression for file-connector for all file input formats
[ https://issues.apache.org/jira/browse/FLINK-33059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot updated FLINK-33059: - Description: Some FileInputFormats don't use FileInputFormat#createSplits (that would detect that the file is non-splittable and deal with reading boundaries correctly), they all create split manually from FileSourceSplit. If input files are compressed, split length is determined by the compressed file length leading to [this|https://issues.apache.org/jira/browse/FLINK-30314] bug. We should force reading the whole file split (like it is done for binary input formats) on compressed files. Parallelism is still done at the file level (as now) (was: Delimited file input formats (contrary to binary input format etc...) do not support compression via the existing decorator because split length is determined by the compressed file length lead to [this|https://issues.apache.org/jira/browse/FLINK-30314] bug . We should force reading the whole file split (like it is done for binary input formats) on compressed files. Parallelism is still done at the file level (as now)) > Support transparent compression for file-connector for all file input formats > - > > Key: FLINK-33059 > URL: https://issues.apache.org/jira/browse/FLINK-33059 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / FileSystem >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > Some FileInputFormats don't use FileInputFormat#createSplits (that would > detect that the file is non-splittable and deal with reading boundaries > correctly), they all create split manually from FileSourceSplit. If input > files are compressed, split length is determined by the compressed file > length leading to [this|https://issues.apache.org/jira/browse/FLINK-30314] > bug. We should force reading the whole file split (like it is done for binary > input formats) on compressed files. Parallelism is still done at the file > level (as now) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint
[
https://issues.apache.org/jira/browse/FLINK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767012#comment-17767012
]
Yordan Pavlov commented on FLINK-33109:
---
Hi [~fanrui] can I ask for your help in testing the latest builds. I have
modified the Dockerfile found at:
https://github.com/apache/flink-docker/blob/master/1.17/scala_2.12-java11-ubuntu/Dockerfile
so that it builds from source code. The branch I am using is `release-1.17`.
Here is the full Dockerfile I am using, in it I am also applying my user Flink
job code.
[https://gist.github.com/YordanPavlov/b9d2f08370dadb5ab18a2dc096b21481]
However, switching a running job (from checkpoint) from the official Flink
1.17.1 image to the one I've built from source gives me the error:
{code:java}
from INITIALIZING to FAILED on 10.42.209.30:44171-33acd8 @
10-42-209-30.xrp-extractor-v4-flink-taskmanager-7.flink.svc.cluster.local
(dataPort=46037).
java.lang.Exception: Exception while creating StreamOperatorStateContext.
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:256)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:256)
at
org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:734)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:709)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state
backend for WindowOperator_439a9d1f894a5a8c69e615cc793a5c7b_(11/32) from any of
the 1 provided restore options.
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:353)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:165)
... 11 common frames omitted
Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught
unexpected exception.
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:407)
at
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:512)
at
org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:99)
at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:336)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
... 13 common frames omitted
Caused by: java.lang.NoSuchMethodError: 'java.util.List
org.apache.flink.runtime.state.IncrementalKeyedStateHandle.getSharedStateHandles()'
at
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restorePreviousIncrementalFilesStatus(RocksDBIncrementalRestoreOperation.java:212)
at
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:188)
at
org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:169)
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:327){code}
Can you hint me in what I might be doing wrong, thanks!
> Watermark alignment not applied after recovery from checkpoint
> --
>
> Key: FLINK-33109
> URL: https://issues.apache.org/jira/browse/FLINK-33109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects
[GitHub] [flink-benchmarks] FangYongs commented on pull request #78: [FLINK-33033][olap][haservice] Add haservice micro benchmark for olap
FangYongs commented on PR #78: URL: https://github.com/apache/flink-benchmarks/pull/78#issuecomment-1727208341 @KarmaGYZ I think there is another purpose to evaluate the performance of OLAP cluster with cluster ZOOKEEPER HA and NONE job HA, and whether it is consistent with the current performance of none HA. After [FLINK-32667](https://issues.apache.org/jira/browse/FLINK-32667), we can keep this benchmark in case of performance degradation caused by other issues -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-32846) [JUnit5 Migration] The metrics, minicluster and net packages of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu resolved FLINK-32846. --- Fix Version/s: 1.19.0 Resolution: Fixed > [JUnit5 Migration] The metrics, minicluster and net packages of flink-runtime > module > > > Key: FLINK-32846 > URL: https://issues.apache.org/jira/browse/FLINK-32846 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Matt Wang >Priority: Minor > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23442: [BP][FLINK-33044][network] Reduce the frequency of triggering flush for the disk tier of the tiered storage
flinkbot commented on PR #23442: URL: https://github.com/apache/flink/pull/23442#issuecomment-1727552325 ## CI report: * 40bead52d200f8aa9cd30336dcc4c3f6f8c68636 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on pull request #23369: [FLINK-33044][network] Reduce the frequency of triggering flush for the disk tier of the tiered storage
TanYuxin-tyx commented on PR #23369: URL: https://github.com/apache/flink/pull/23369#issuecomment-1727044233 @xintongsong @reswqa Thanks for review the change. I update the code to rebase the fixup commit and trigger the tests again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33125) Upgrade JOSDK to 4.4.4
Nicolas Fraison created FLINK-33125: --- Summary: Upgrade JOSDK to 4.4.4 Key: FLINK-33125 URL: https://issues.apache.org/jira/browse/FLINK-33125 Project: Flink Issue Type: Bug Components: Kubernetes Operator Reporter: Nicolas Fraison Fix For: kubernetes-operator-1.7.0 JOSDK [4.4.4|https://github.com/operator-framework/java-operator-sdk/releases/tag/v4.4.4] contains fix for leader election issue we face in our environment Here are more information on the [issue|https://github.com/operator-framework/java-operator-sdk/issues/2056] faced -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33126) Fix EventTimeAllWindowCheckpointingITCase jobName typo
[ https://issues.apache.org/jira/browse/FLINK-33126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu reassigned FLINK-33126: - Assignee: Yue Ma (was: Yue Ma) > Fix EventTimeAllWindowCheckpointingITCase jobName typo > -- > > Key: FLINK-33126 > URL: https://issues.apache.org/jira/browse/FLINK-33126 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.1 >Reporter: Yue Ma >Assignee: Yue Ma >Priority: Minor > > Fix EventTimeAllWindowCheckpointingITCase jobName Typo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33110) Array content gets replaced with last element duplicates
[
https://issues.apache.org/jira/browse/FLINK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767106#comment-17767106
]
Martijn Visser commented on FLINK-33110:
[~romanlupiichuk] Can you please verify if this issue still exists with Flink
1.17?
> Array content gets replaced with last element duplicates
>
>
> Key: FLINK-33110
> URL: https://issues.apache.org/jira/browse/FLINK-33110
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.15.4
>Reporter: Roman Lupiichuk
>Priority: Minor
>
> After upgrade from Flink 1.14.0 to 1.15.4 a bunch of our tests started to
> fail.
> I've striped down one of the failing test to following (it's in Kotlin)
> {code:java}
> import org.apache.flink.configuration.Configuration
> import org.apache.flink.table.annotation.DataTypeHint
> import org.apache.flink.table.annotation.FunctionHint
> import org.apache.flink.table.annotation.InputGroup
> import org.apache.flink.table.api.TableEnvironment
> import org.apache.flink.table.functions.ScalarFunction
> import org.apache.flink.table.planner.factories.TestValuesTableFactory
> import org.apache.flink.types.Row
> import org.junit.jupiter.api.Test
> @FunctionHint(output = DataTypeHint("ARRAY>"))
> object TestArrayFunc : ScalarFunction() {
> fun eval(@DataTypeHint(inputGroup = InputGroup.ANY) vararg values: Any):
> Array =
> values
> .map { data ->
> val casted = data as Map
> Row.of(casted["fieldName"])
> }
> .toTypedArray()
> }
> class ArrayFieldTest {
> @Test
> fun test() {
> val tableEnv = TableEnvironment.create(
> Configuration().also {
> it.setString("table.exec.resource.default-parallelism", "1")
> },
> )
> tableEnv.createTemporarySystemFunction("TO_FIELDS_ARRAY",
> TestArrayFunc)
> val dataId = TestValuesTableFactory.registerData(
> listOf(
> TestValuesTableFactory.changelogRow(
> "+I",
> "123"
> )
> )
> )
> tableEnv.executeSql(
> """
> CREATE TABLE events
> (
> id STRING
> ) WITH (
> 'connector' = 'values',
> 'data-id' = '$dataId'
> )
> """
> )
> tableEnv.executeSql(
> """
> CREATE TABLE results
> (
> fields ARRAY>,
> event_time TIMESTAMP
> ) WITH (
> 'connector' = 'print'
> )
> """
> )
> tableEnv.executeSql(
> """
> INSERT INTO results (fields, event_time)
> SELECT
> TO_FIELDS_ARRAY(
>MAP['fieldName', 'foo'],
>MAP['fieldName', 'hello']
> ),
> NOW()
> FROM events
> """
> )
> }
> }
> {code}
> In Flink 1.14.0 it produces
> {code:java}
> +I[[+I[foo], +I[hello]], 2023-09-18T08:18:55.278]{code}
> That's correct and expected output.
> But in Flink 1.15.4 the output is
> {code:java}
> +I[[+I[hello], +I[hello]], 2023-09-18T08:21:12.569]{code}
> As one can see all elements in the array were replaced with the last element
> duplicates.
> The issue goes away if I
> # either remove NOT NULL constraint from function hint
> # or remove TIMESTAMP field from the sink table
> There is also no issue in regular Flink cluster, only in MiniCluster which is
> used in testing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23441: [FLINK-33060] Fix the javadoc of ListState interfaces about not allowing null value
flinkbot commented on PR #23441: URL: https://github.com/apache/flink/pull/23441#issuecomment-1727408582 ## CI report: * 99f4dd9965ba131ba6abceada5a9969555fbb033 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23445: [FLINK-15736][docs] Add Java compatibility page
flinkbot commented on PR #23445: URL: https://github.com/apache/flink/pull/23445#issuecomment-1727692469 ## CI report: * 9ae7f3e45446f34d806ea6a98e6aa5be548b2b78 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-30314) Unable to read all records from compressed delimited file input format
[
https://issues.apache.org/jira/browse/FLINK-30314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767099#comment-17767099
]
Etienne Chauchot commented on FLINK-30314:
--
https://github.com/apache/flink/pull/23443
> Unable to read all records from compressed delimited file input format
> --
>
> Key: FLINK-30314
> URL: https://issues.apache.org/jira/browse/FLINK-30314
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.16.0, 1.15.2, 1.17.1
>Reporter: Dmitry Yaraev
>Assignee: Etienne Chauchot
>Priority: Major
> Attachments: input.json, input.json.gz, input.json.zip
>
>
> I am reading gzipped JSON line-delimited files in the batch mode using
> [FileSystem
> Connector|https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/filesystem/].
> For reading the files a new table is created with the following
> configuration:
> {code:sql}
> CREATE TEMPORARY TABLE `my_database`.`my_table` (
> `my_field1` BIGINT,
> `my_field2` INT,
> `my_field3` VARCHAR(2147483647)
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = 'path-to-input-dir',
> 'format' = 'json',
> 'json.ignore-parse-errors' = 'false',
> 'json.fail-on-missing-field' = 'true'
> ) {code}
> In the input directory I have two files: input-0.json.gz and
> input-1.json.gz. As it comes from the filenames, the files are compressed
> with GZIP. Each of the files contains 10 records. The issue is that only 2
> records from each file are read (4 in total). If decompressed versions of the
> same data files are used, all 20 records are read.
> As far as I understand, that problem may be related to the fact that split
> length, which is used when the files are read, is in fact the length of a
> compressed file. So files are closed before all records are read from them
> because read position of the decompressed file stream exceeds split length.
> Probably, it makes sense to add a flag to {{{}FSDataInputStream{}}}, so we
> could identify if the file compressed or not. The flag can be set to true in
> {{InputStreamFSInputWrapper}} because it is used for wrapping compressed file
> streams. With such a flag it could be possible to differentiate
> non-splittable compressed files and only rely on the end of the stream.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32846) [JUnit5 Migration] The metrics, minicluster and net packages of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767056#comment-17767056 ] Weihua Hu commented on FLINK-32846: --- Metrics merged in master: 5a8321f6385f9a108773989a4ca176af1f3c72d4 > [JUnit5 Migration] The metrics, minicluster and net packages of flink-runtime > module > > > Key: FLINK-32846 > URL: https://issues.apache.org/jira/browse/FLINK-32846 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Matt Wang >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32197) FLIP 246: Dynamic Kafka Source
[ https://issues.apache.org/jira/browse/FLINK-32197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767055#comment-17767055 ] Yun Tang commented on FLINK-32197: -- [~mason6345] Thanks for the information! I will also ask some guys to take a look of the PR. > FLIP 246: Dynamic Kafka Source > -- > > Key: FLINK-32197 > URL: https://issues.apache.org/jira/browse/FLINK-32197 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka >Affects Versions: kafka-3.1.0 >Reporter: Mason Chen >Assignee: Mason Chen >Priority: Major > Fix For: kafka-3.1.0 > > > This is for introducing a new connector that extends off the current > KafkaSource to read multiple Kafka clusters, which can change dynamically. > For more details, please refer to [FLIP > 246|https://cwiki.apache.org/confluence/display/FLINK/FLIP-246%3A+Multi+Cluster+Kafka+Source]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] TanYuxin-tyx opened a new pull request, #23442: [BP][FLINK-33044][network] Reduce the frequency of triggering flush for the disk tier of the tiered storage
TanYuxin-tyx opened a new pull request, #23442: URL: https://github.com/apache/flink/pull/23442 ## What is the purpose of the change *Backport the pr https://github.com/apache/flink/pull/23369 to release 1.18.* ## Brief change log - *Backport the pr https://github.com/apache/flink/pull/23369 to release 1.18.* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a backport without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-33126) Fix EventTimeAllWindowCheckpointingITCase jobName typo
[ https://issues.apache.org/jira/browse/FLINK-33126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weihua Hu reassigned FLINK-33126: - Assignee: Yue Ma > Fix EventTimeAllWindowCheckpointingITCase jobName typo > -- > > Key: FLINK-33126 > URL: https://issues.apache.org/jira/browse/FLINK-33126 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.1 >Reporter: Yue Ma >Assignee: Yue Ma >Priority: Minor > > Fix EventTimeAllWindowCheckpointingITCase jobName Typo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23444: [FLINK-33126] Fix EventTimeAllWindowCheckpointingITCase jobName typo
flinkbot commented on PR #23444: URL: https://github.com/apache/flink/pull/23444#issuecomment-1727680576 ## CI report: * d6bfd73e33c038e4b3f66f15b7f8b75b69a651a3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #23443: [FLINK-33059] Support transparent compression for file-connector for all file input formats
flinkbot commented on PR #23443: URL: https://github.com/apache/flink/pull/23443#issuecomment-1727591627 ## CI report: * f5e4b88c3c51e1759e42cc06373fa608cb3bd1f9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-32197) FLIP 246: Dynamic Kafka Source
[ https://issues.apache.org/jira/browse/FLINK-32197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767055#comment-17767055 ] Yun Tang edited comment on FLINK-32197 at 9/20/23 8:59 AM: --- [~mason6345] Thanks for the information! I will also ask some guys to take a look of the PR when possible. was (Author: yunta): [~mason6345] Thanks for the information! I will also ask some guys to take a look of the PR. > FLIP 246: Dynamic Kafka Source > -- > > Key: FLINK-32197 > URL: https://issues.apache.org/jira/browse/FLINK-32197 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka >Affects Versions: kafka-3.1.0 >Reporter: Mason Chen >Assignee: Mason Chen >Priority: Major > Fix For: kafka-3.1.0 > > > This is for introducing a new connector that extends off the current > KafkaSource to read multiple Kafka clusters, which can change dynamically. > For more details, please refer to [FLIP > 246|https://cwiki.apache.org/confluence/display/FLINK/FLIP-246%3A+Multi+Cluster+Kafka+Source]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage
[ https://issues.apache.org/jira/browse/FLINK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weijie Guo updated FLINK-33044: --- Fix Version/s: 1.19.0 > Reduce the frequency of triggering flush for the disk tier of the tiered > storage > > > Key: FLINK-33044 > URL: https://issues.apache.org/jira/browse/FLINK-33044 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Fix For: 1.19.0 > > > The disk cache of tiered storage will flush at the end of each subpartition's > segment, which is too frequent and is bad for performance. We should improve > it with some better flushing methods, e.g. flushing buffers with batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] reswqa merged pull request #23369: [FLINK-33044][network] Reduce the frequency of triggering flush for the disk tier of the tiered storage
reswqa merged PR #23369: URL: https://github.com/apache/flink/pull/23369 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler
[
https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhanghao Chen updated FLINK-33123:
--
Summary: Wrong dynamic replacement of partitioner from FORWARD to REBLANCE
for autoscaler and adaptive scheduler (was: Wrong dynamic replacement of
partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and
)
> Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for
> autoscaler and adaptive scheduler
> ---
>
> Key: FLINK-33123
> URL: https://issues.apache.org/jira/browse/FLINK-33123
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler, Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Zhanghao Chen
>Priority: Critical
> Attachments: image-2023-09-20-15-09-22-733.png,
> image-2023-09-20-15-14-04-679.png
>
>
> *Background*
> https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
> wrong when the parallelism is changed for a vertex with a FORWARD edge, which
> is used by both the autoscaler and adaptive scheduler where one can change
> the vertex parallelism dynamically. Fix is applied to dynamically replace
> partitioner from FORWARD to REBLANCE on task deployment in
> {{{}StreamTask{}}}:
>
> !image-2023-09-20-15-09-22-733.png|width=560,height=221!
> *Problem*
> Unfortunately, the fix is still buggy in two aspects:
> # The connections between upstream and downstream tasks are determined by
> the distribution type of the partitioner when generating execution graph on
> the JM side. When the edge is FORWARD, the distribution type is POINTWISE,
> and Flink will try to evenly distribute subpartitions to all downstream
> tasks. If one want to change it to REBALANCE, the distribution type has to be
> changed to ALL_TO_ALL to make all-to-all connections between upstream and
> downstream tasks. However, the fix did not change the distribution type which
> makes the network connections be set up in a wrong way.
> # The FOWARD partitioner will be replaced if
> environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
> task parallelism. However, the number of subpartitions here equals to the
> number of downstream tasks of this particular task, which is also determined
> by the distribution type of the partitioner when generating execution graph
> on the JM side. When ceil(downstream task parallelism / upstream task
> parallelism) = upstream task parallelism, we will have the number of
> subpartitions = task parallelism. For example, for a topology A (parallelism
> 2) -> B (parallelism 5), we will have 1 A task having 2 subpartitions, 1 A
> task having 3 subpartition, and hence 1 task will have its number of
> subpartitions equals to the task parallelism 2 and skip partitioner
> replacement. As a result, that task will only send data to only one
> downstream task as the FORWARD partitioner always send data to the first
> subpartition. In fact, for a normal job with a FORWARD edge without any
> autoscaling action, you will find that the partitioner is changed to
> REBALANCE internally as the number of subpartitions always equals to 1 in
> this case.
> !image-2023-09-20-15-14-04-679.png|width=892,height=301!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and
[
https://issues.apache.org/jira/browse/FLINK-33123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767035#comment-17767035
]
Zhanghao Chen commented on FLINK-33123:
---
Hi [~gyfora] [~mxm] , please help take a look. I personally think that
dynamically replacing partitioner on task deployment time is a wrong direction
to follow and we'd better dynamically change the partitioner info on the
jobgraph side. However, the currently way of encoding partitioner info makes it
difficult to do so and may require a large-scale refactoring. Maybe we can
apply a quick fix first and consider the refactoring later:
* If we just want to change FORWARD to RESCALE, the easiest fix would be to
always replace FORWARD partitioner by RESCALE partitioner in StreamTask. When
the number of subpartitions = 1, the behavior of FORWARD and RESCALE
partitioner is actually the same.
* If we just want to change FORWARD to REBALANCE, the easiest fix would be to
always replace FORWARD partitioner by REBALANCE partitioner in StreamTask +
changing the distribution type to ALL_TO_ALL in jobgraph.
Looking forward to your opinions on it.
> Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for
> autoscaler and adaptive scheduler and
> -
>
> Key: FLINK-33123
> URL: https://issues.apache.org/jira/browse/FLINK-33123
> Project: Flink
> Issue Type: Bug
> Components: Autoscaler, Runtime / Coordination
>Affects Versions: 1.17.0, 1.18.0
>Reporter: Zhanghao Chen
>Priority: Critical
> Attachments: image-2023-09-20-15-09-22-733.png,
> image-2023-09-20-15-14-04-679.png
>
>
> *Background*
> https://issues.apache.org/jira/browse/FLINK-30213 reported that the edge is
> wrong when the parallelism is changed for a vertex with a FORWARD edge, which
> is used by both the autoscaler and adaptive scheduler where one can change
> the vertex parallelism dynamically. Fix is applied to dynamically replace
> partitioner from FORWARD to REBLANCE on task deployment in
> {{{}StreamTask{}}}:
>
> !image-2023-09-20-15-09-22-733.png|width=560,height=221!
> *Problem*
> Unfortunately, the fix is still buggy in two aspects:
> # The connections between upstream and downstream tasks are determined by
> the distribution type of the partitioner when generating execution graph on
> the JM side. When the edge is FORWARD, the distribution type is POINTWISE,
> and Flink will try to evenly distribute subpartitions to all downstream
> tasks. If one want to change it to REBALANCE, the distribution type has to be
> changed to ALL_TO_ALL to make all-to-all connections between upstream and
> downstream tasks. However, the fix did not change the distribution type which
> makes the network connections be set up in a wrong way.
> # The FOWARD partitioner will be replaced if
> environment.getWriter(outputIndex).getNumberOfSubpartitions() equals to the
> task parallelism. However, the number of subpartitions here equals to the
> number of downstream tasks of this particular task, which is also determined
> by the distribution type of the partitioner when generating execution graph
> on the JM side. When ceil(downstream task parallelism / upstream task
> parallelism) = upstream task parallelism, we will have the number of
> subpartitions = task parallelism. For example, for a topology A (parallelism
> 2) -> B (parallelism 5), we will have 1 A task having 2 subpartitions, 1 A
> task having 3 subpartition, and hence 1 task will have its number of
> subpartitions equals to the task parallelism 2 and skip partitioner
> replacement. As a result, that task will only send data to only one
> downstream task as the FORWARD partitioner always send data to the first
> subpartition. In fact, for a normal job with a FORWARD edge without any
> autoscaling action, you will find that the partitioner is changed to
> REBALANCE internally as the number of subpartitions always equals to 1 in
> this case.
> !image-2023-09-20-15-14-04-679.png|width=892,height=301!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33044) Reduce the frequency of triggering flush for the disk tier of the tiered storage
[ https://issues.apache.org/jira/browse/FLINK-33044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767089#comment-17767089 ] Weijie Guo commented on FLINK-33044: master(1.19) via b076c52d8da914e81c3e004c0b0c7883463bb151. > Reduce the frequency of triggering flush for the disk tier of the tiered > storage > > > Key: FLINK-33044 > URL: https://issues.apache.org/jira/browse/FLINK-33044 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > > The disk cache of tiered storage will flush at the end of each subpartition's > segment, which is too frequent and is bad for performance. We should improve > it with some better flushing methods, e.g. flushing buffers with batch. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33124) Kafka Connector not working for table
Aarsh Shah created FLINK-33124: -- Summary: Kafka Connector not working for table Key: FLINK-33124 URL: https://issues.apache.org/jira/browse/FLINK-33124 Project: Flink Issue Type: Bug Reporter: Aarsh Shah -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol opened a new pull request, #23445: [FLINK-15736][docs] Add Java compatibility page
zentol opened a new pull request, #23445: URL: https://github.com/apache/flink/pull/23445 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot opened a new pull request, #23443: [Flink-33059] Support transparent compression for file-connector for all file input formats
echauchot opened a new pull request, #23443: URL: https://github.com/apache/flink/pull/23443 ## What is the purpose of the change Support transparent compression for file-connector for all file input formats. ## Brief change log - Force reading the whole file split (like it is done for binary input formats) on compressed (unsplittable) files - add FileInputFormatTest#testFileInputFormatWithCompressionFromFileSource - generalize FileInputFormatTest#testFileInputFormatWithCompression to more that deflate format ## Verifying this change FileInputFormatTest#testFileInputFormatWithCompressionFromFileSource ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? website -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] mayuehappy opened a new pull request, #23444: [FLINK-33126] Fix EventTimeAllWindowCheckpointingITCase jobName typo
mayuehappy opened a new pull request, #23444: URL: https://github.com/apache/flink/pull/23444 ## What is the purpose of the change Fix EventTimeAllWindowCheckpointingITCase jobName typo ## Brief change log Adjust the jobName of test in EventTimeAllWindowCheckpointingITCase ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33111) Flink Pulsar Connector to Pulsar Client Version Mismatch
[ https://issues.apache.org/jira/browse/FLINK-33111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767104#comment-17767104 ] Martijn Visser commented on FLINK-33111: [~Tison] WDYT? > Flink Pulsar Connector to Pulsar Client Version Mismatch > > > Key: FLINK-33111 > URL: https://issues.apache.org/jira/browse/FLINK-33111 > Project: Flink > Issue Type: Bug > Components: Connectors / Pulsar >Affects Versions: 1.17.1 >Reporter: Jason Kania >Priority: Major > > In the documentation for the Flink Pulsar Connector, > ([https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/pulsar/]) > it indicates that 2.10.0 and above versions of the pulsar client are > supported "You can use the connector with the Pulsar 2.10.0 or higher" and > the pom file entry references the 4.0.0-1.17 version of the connector which > points to the 2.11.0 version of the Pulsar client. However, when using Pulsar > Client 2.10.4 or 2.10.5, the following error is generated: > > java.lang.NoSuchMethodError: 'org.apache.pulsar.client.api.ClientBuilder > org.apache.pulsar.client.api.ClientBuilder.connectionMaxIdleSeconds(int)' > at > org.apache.flink.connector.pulsar.common.config.PulsarClientFactory.createClient(PulsarClientFactory.java:127) > at > org.apache.flink.connector.pulsar.source.reader.PulsarSourceReader.create(PulsarSourceReader.java:266) > at > org.apache.flink.connector.pulsar.source.PulsarSource.createReader(PulsarSource.java:137) > at > org.apache.flink.streaming.api.operators.SourceOperator.initReader(SourceOperator.java:312) > at > org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask.init(SourceOperatorStreamTask.java:93) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:699) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:675) > at > org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) > at > org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:921) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) > at java.base/java.lang.Thread.run(Thread.java:829) > > The referenced method 'connectionMaxIdleSeconds' is only available in the > Pulsar 2.11 client when looking at the source code. I am not sure whether the > documentation is wrong and the Flink Pulsar Connector 2.11 is the intended > Pulsar version. However, my understanding is that Pulsar 2.11 is targeted > toward java 17. This would create the need for mixed Java 11 and Java 17 > deployment unless the Pulsar client code is compiled for 2.11. > > Documentation cleanup and a reference to the appropriate Java versions is > needed. A fix to the 1.17.1 Flink pulsar connector may alternatively be > required. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33126) Fix EventTimeAllWindowCheckpointingITCase jobName typo
Yue Ma created FLINK-33126: -- Summary: Fix EventTimeAllWindowCheckpointingITCase jobName typo Key: FLINK-33126 URL: https://issues.apache.org/jira/browse/FLINK-33126 Project: Flink Issue Type: Improvement Components: Tests Affects Versions: 1.17.1 Reporter: Yue Ma Fix EventTimeAllWindowCheckpointingITCase jobName Typo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] 911432 closed pull request #23380: Readme.md changes according to [FLINK-25002]
911432 closed pull request #23380: Readme.md changes according to [FLINK-25002] URL: https://github.com/apache/flink/pull/23380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20767) add nested field support for SupportsFilterPushDown
[ https://issues.apache.org/jira/browse/FLINK-20767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-20767: --- Fix Version/s: (was: 1.18.0) > add nested field support for SupportsFilterPushDown > --- > > Key: FLINK-20767 > URL: https://issues.apache.org/jira/browse/FLINK-20767 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.12.0 >Reporter: Jun Zhang >Priority: Major > Labels: pull-request-available > > I think we should add the nested field support for SupportsFilterPushDown -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33052) codespeed and benchmark server is down
[ https://issues.apache.org/jira/browse/FLINK-33052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17767032#comment-17767032 ] Zakelly Lan commented on FLINK-33052: - Hi [~pnowojski] [~jingge] , Is it possible we change the original domain name (codespeed.dak8s.net) resolution to the new server? Or I would rather buy a more official domain name like 'flink-speed.io' for the new speed center website. WDTY? > codespeed and benchmark server is down > -- > > Key: FLINK-33052 > URL: https://issues.apache.org/jira/browse/FLINK-33052 > Project: Flink > Issue Type: Bug > Components: Benchmarks, Test Infrastructure >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Zakelly Lan >Priority: Blocker > > No update in #flink-dev-benchmarks slack channel since 25th August. > It was a EC2 running in a legacy aws account. Currently on one knows which > account it is. > > https://apache-flink.slack.com/archives/C0471S0DFJ9/p1693932155128359 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33126) Fix EventTimeAllWindowCheckpointingITCase jobName typo
[ https://issues.apache.org/jira/browse/FLINK-33126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yue Ma updated FLINK-33126: --- Issue Type: Bug (was: Improvement) > Fix EventTimeAllWindowCheckpointingITCase jobName typo > -- > > Key: FLINK-33126 > URL: https://issues.apache.org/jira/browse/FLINK-33126 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.1 >Reporter: Yue Ma >Priority: Minor > > Fix EventTimeAllWindowCheckpointingITCase jobName Typo -- This message was sent by Atlassian Jira (v8.20.10#820010)
