[jira] [Assigned] (FLINK-34102) Invalid configuration when using 'env.log.max' on yarn application mode
[ https://issues.apache.org/jira/browse/FLINK-34102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-34102: --- Assignee: RocMarshal > Invalid configuration when using 'env.log.max' on yarn application mode > --- > > Key: FLINK-34102 > URL: https://issues.apache.org/jira/browse/FLINK-34102 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.19.0 >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33988][configuration] Fix the invalid configuration when using initialized root logger level on yarn deployment mode [flink]
1996fanrui commented on code in PR #24042:
URL: https://github.com/apache/flink/pull/24042#discussion_r1453038598
##
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java:
##
@@ -216,6 +229,24 @@ public YarnClusterDescriptor(
this.nodeLabel =
flinkConfiguration.getString(YarnConfigOptions.NODE_LABEL);
}
+/** Adapt flink env setting. */
+private static void adaptEnvSetting(
+Configuration config,
+ConfigOption configOption,
+String envKey,
+Function validator) {
Review Comment:
As I understand, this `validator` wanna check whether the value of
`configOption` in config is legal.
- If it's legal, we can add the env
- If it's illegal, we will ignore.
Is my understanding right?
If right, I don't think we should do this check here. I mean we should cover
Flink on Kubernetes if we wanna check whether it's legal. So check them in the
yarn side doesn't make sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [revert][FLINK-33907][ci] Makes copying test jars being done in the package phase [flink]
XComp commented on PR #24095: URL: https://github.com/apache/flink/pull/24095#issuecomment-1893225487 I responded in the [FLINK-33907](https://issues.apache.org/jira/browse/FLINK-33907?focusedCommentId=17807087=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17807087) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34106) user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider
[ https://issues.apache.org/jira/browse/FLINK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807088#comment-17807088 ] Zhanghao Chen commented on FLINK-34106: --- [~Akihito Liang] Are you refering to the new API introduced in [FLIP-367|[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150]?] If so, it is still undergoing, but we'll make it in Flink 1.19. Please subscribe FLINK-33261 for the latest progress > user defined source parallelism in ddl can't work if connector > ScanRuntimeProvider is SourceFunctionProvider > > > Key: FLINK-34106 > URL: https://issues.apache.org/jira/browse/FLINK-34106 > Project: Flink > Issue Type: Bug >Reporter: Junning Liang >Priority: Critical > > I implemented a connector using the SourceFunctionProvider myself and found > that setting scan parallelism is not effective. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34106) user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider
[ https://issues.apache.org/jira/browse/FLINK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807088#comment-17807088 ] Zhanghao Chen edited comment on FLINK-34106 at 1/16/24 7:54 AM: [~Akihito Liang] Are you refering to the new API introduced in [FLIP-367|[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150]]? If so, it is still undergoing, but we'll make it in Flink 1.19. Please subscribe FLINK-33261 for the latest progress was (Author: zhanghao chen): [~Akihito Liang] Are you refering to the new API introduced in [FLIP-367|[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150]?] If so, it is still undergoing, but we'll make it in Flink 1.19. Please subscribe FLINK-33261 for the latest progress > user defined source parallelism in ddl can't work if connector > ScanRuntimeProvider is SourceFunctionProvider > > > Key: FLINK-34106 > URL: https://issues.apache.org/jira/browse/FLINK-34106 > Project: Flink > Issue Type: Bug >Reporter: Junning Liang >Priority: Critical > > I implemented a connector using the SourceFunctionProvider myself and found > that setting scan parallelism is not effective. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34106) user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider
[ https://issues.apache.org/jira/browse/FLINK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807088#comment-17807088 ] Zhanghao Chen edited comment on FLINK-34106 at 1/16/24 7:54 AM: [~Akihito Liang] Are you refering to the new API introduced in [FLIP-367|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150]? If so, it is still undergoing, but we'll make it in Flink 1.19. Please subscribe FLINK-33261 for the latest progress was (Author: zhanghao chen): [~Akihito Liang] Are you refering to the new API introduced in [FLIP-367|[https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429150]]? If so, it is still undergoing, but we'll make it in Flink 1.19. Please subscribe FLINK-33261 for the latest progress > user defined source parallelism in ddl can't work if connector > ScanRuntimeProvider is SourceFunctionProvider > > > Key: FLINK-34106 > URL: https://issues.apache.org/jira/browse/FLINK-34106 > Project: Flink > Issue Type: Bug >Reporter: Junning Liang >Priority: Critical > > I implemented a connector using the SourceFunctionProvider myself and found > that setting scan parallelism is not effective. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33907) Makes copying test jars being done later
[
https://issues.apache.org/jira/browse/FLINK-33907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807087#comment-17807087
]
Matthias Pohl commented on FLINK-33907:
---
I had a discussion with [~chesnay] on that issue. His argument from the comment
above is reasonable. We would need to improve the Maven setup to make the
individual test execution work in this specific case (either through the
utilization of the failsafe plugin or by introducing another profile). This is
covered by FLINK-34107. There is a (admittedly not so convenient) workaround
documented in FLINK-34107. Alternatively, you can run {{mvn -pl flink-client
verify}}.
The actual goal of FLINK-33907 is to allow for an improved performance of the
build pipeline. That outweighs the need to run two tests individually on the
command line in my opinion. [~gyfora] WDYT?
> Makes copying test jars being done later
>
>
> Key: FLINK-33907
> URL: https://issues.apache.org/jira/browse/FLINK-33907
> Project: Flink
> Issue Type: Sub-task
> Components: Build System / CI
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: github-actions, pull-request-available
> Fix For: 1.19.0, 1.18.1
>
>
> We experienced an issue in GHA which is due to the fact how test resources
> are pre-computed in GHA:
> {code:java}
> This fixes the following error when compiling flink-clients:
> Error: 2.054 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:3.2.0:copy-dependencies
> (copy-dependencies) on project flink-clients: Artifact has not been packaged
> yet. When used on reactor artifact, copy should be executed after packaging:
> see MDEP-187. -> [Help 1] {code}
> We need to move this goal to a later phase.
> The reason why this popped up is (as far as I remember) that we do only do
> test-compile in GitHub Actions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33988][configuration] Fix the invalid configuration when using initialized root logger level on yarn deployment mode [flink]
1996fanrui commented on code in PR #24042:
URL: https://github.com/apache/flink/pull/24042#discussion_r1453038598
##
flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java:
##
@@ -216,6 +229,24 @@ public YarnClusterDescriptor(
this.nodeLabel =
flinkConfiguration.getString(YarnConfigOptions.NODE_LABEL);
}
+/** Adapt flink env setting. */
+private static void adaptEnvSetting(
+Configuration config,
+ConfigOption configOption,
+String envKey,
+Function validator) {
Review Comment:
As I understand, this `validator` wanna check whether the value of
`configOption` in config is legal.
- If it's legal, we can add the env
- If it's illegal, we will ignore.
Is my understanding right?
If right, I don't think we should do this check here. I means we should
cover Flink on Kubernetes if we wanna check whether it's legal. So check them
in the yarn side doesn't make sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34107) Clearer separation of unit test and integration tests
[
https://issues.apache.org/jira/browse/FLINK-34107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807081#comment-17807081
]
Matthias Pohl commented on FLINK-34107:
---
One approach here could be to use the [failsafe
plugin|https://maven.apache.org/surefire/maven-failsafe-plugin/] for the
integration test execution. This wasn't done so far to keep the number of
plugins in the Flink build low (it was conveniently enough to have the test
executions being covered by the surefire plugin).
> Clearer separation of unit test and integration tests
> -
>
> Key: FLINK-34107
> URL: https://issues.apache.org/jira/browse/FLINK-34107
> Project: Flink
> Issue Type: Bug
> Components: Build System, Build System / CI
>Affects Versions: 1.18.0, 1.17.2, 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
>
> FLINK-33907 revealed an issue in the current Maven setup where the
> surefire-plugin is in charge of executing both, the unit and the integration
> tests in two separate executions: The {{-Dtest}} system parameter is
> overwriting the {{}} and {{}} configuration of the two
> executions. That results in a test being executed twice if specifically
> selected for execution (i.e. {{mvn [...] verify -Dtest=???}}).
> That's especially problematic in cases where the ITCase relies on artifacts
> that are build during the Maven run (see the discussion in FLINK-33907 where
> {{DefaultPackagedProgramRetrieverITCase}} relies on
> {{flink-client-test-utils}} to provide test jars). The artifacts should be
> moved in the {{pre-integration-test}} phase from a conceptual point of view.
> A workaround for this issue is to directly call the execution
> {{surefire:test@integration-tests}}:
> {code}
> mvn -pl flink-clients surefire:test@integration-tests
> -Dtest=DefaultPackagedProgramRetrieverITCase -Dfast
> {code}
> This approach has the flow that the Maven lifecycle is ignored. You would
> have to do a {{mvn -pl flink-clients clean verify}} run upfront to build all
> the artifacts the test relies on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34107) Clearer separation of unit test and integration tests
Matthias Pohl created FLINK-34107:
-
Summary: Clearer separation of unit test and integration tests
Key: FLINK-34107
URL: https://issues.apache.org/jira/browse/FLINK-34107
Project: Flink
Issue Type: Bug
Components: Build System, Build System / CI
Affects Versions: 1.17.2, 1.18.0, 1.19.0
Reporter: Matthias Pohl
FLINK-33907 revealed an issue in the current Maven setup where the
surefire-plugin is in charge of executing both, the unit and the integration
tests in two separate executions: The {{-Dtest}} system parameter is
overwriting the {{}} and {{}} configuration of the two
executions. That results in a test being executed twice if specifically
selected for execution (i.e. {{mvn [...] verify -Dtest=???}}).
That's especially problematic in cases where the ITCase relies on artifacts
that are build during the Maven run (see the discussion in FLINK-33907 where
{{DefaultPackagedProgramRetrieverITCase}} relies on {{flink-client-test-utils}}
to provide test jars). The artifacts should be moved in the
{{pre-integration-test}} phase from a conceptual point of view.
A workaround for this issue is to directly call the execution
{{surefire:test@integration-tests}}:
{code}
mvn -pl flink-clients surefire:test@integration-tests
-Dtest=DefaultPackagedProgramRetrieverITCase -Dfast
{code}
This approach has the flow that the Maven lifecycle is ignored. You would have
to do a {{mvn -pl flink-clients clean verify}} run upfront to build all the
artifacts the test relies on.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
snuyanzin commented on code in PR #23411:
URL: https://github.com/apache/flink/pull/23411#discussion_r1452991614
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/ArrayAggFunctionITCase.java:
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.INT;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.types.RowKind.DELETE;
+import static org.apache.flink.types.RowKind.INSERT;
+import static org.apache.flink.types.RowKind.UPDATE_AFTER;
+import static org.apache.flink.types.RowKind.UPDATE_BEFORE;
+
+/** Tests for built-in ARRAY_AGG aggregation functions. */
+class ArrayAggFunctionITCase extends BuiltInAggregateFunctionTestBase {
+
+@Override
+Stream getTestCaseSpecs() {
+return Stream.of(
+TestSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_AGG)
+.withDescription("ARRAY changelog stream aggregation")
+.withSource(
+ROW(STRING(), INT()),
+Arrays.asList(
+Row.ofKind(INSERT, "A", 1),
+Row.ofKind(INSERT, "A", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 3),
+Row.ofKind(INSERT, "C", 3),
+Row.ofKind(INSERT, "C", null),
+Row.ofKind(INSERT, "D", null),
+Row.ofKind(INSERT, "E", 4),
+Row.ofKind(INSERT, "E", 5),
+Row.ofKind(DELETE, "E", 5),
+Row.ofKind(UPDATE_BEFORE, "E", 4),
+Row.ofKind(UPDATE_AFTER, "E", 6)))
+.testResult(
+source ->
+"SELECT f0, array_agg(f1) FROM " +
source + " GROUP BY f0",
+TableApiAggSpec.groupBySelect(
+Collections.singletonList($("f0")),
+$("f0"),
+$("f1").arrayAgg()),
+ROW(STRING(), ARRAY(INT())),
+ROW(STRING(), ARRAY(INT())),
+Arrays.asList(
+Row.of("A", new Integer[] {1, 2}),
+Row.of("B", new Integer[] {2, 2, 3}),
+Row.of("C", new Integer[] {3}),
Review Comment:
It's ok to support it only partially however the main concern here:
~~I would expect more consistent result
since there is also input `Row.ofKind(INSERT, "D", null),` and expected
value `Row.of("D", null),`
So I would expect for ths either containing nulls or both not containing
null however not mixed~~
UPD: to be more clear:
after playing with Postgres and BigQuery I noticed that both respect nulls
by default and for BigQuery to make it ignoring nulls it should be specified
explicitely. Is there a reason why for Flink it is done differently? Probably
it's better to have similar behaviour
I don't tell that we need to support `RESPECT NULLS | IGNORE NULLS` syntax,
however need to make `RESPECT NULLS` default behaviour to be on same page with
vendors
--
This is an automated message from the Apache Git
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
snuyanzin commented on code in PR #23411:
URL: https://github.com/apache/flink/pull/23411#discussion_r1452991614
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/ArrayAggFunctionITCase.java:
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.INT;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.types.RowKind.DELETE;
+import static org.apache.flink.types.RowKind.INSERT;
+import static org.apache.flink.types.RowKind.UPDATE_AFTER;
+import static org.apache.flink.types.RowKind.UPDATE_BEFORE;
+
+/** Tests for built-in ARRAY_AGG aggregation functions. */
+class ArrayAggFunctionITCase extends BuiltInAggregateFunctionTestBase {
+
+@Override
+Stream getTestCaseSpecs() {
+return Stream.of(
+TestSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_AGG)
+.withDescription("ARRAY changelog stream aggregation")
+.withSource(
+ROW(STRING(), INT()),
+Arrays.asList(
+Row.ofKind(INSERT, "A", 1),
+Row.ofKind(INSERT, "A", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 3),
+Row.ofKind(INSERT, "C", 3),
+Row.ofKind(INSERT, "C", null),
+Row.ofKind(INSERT, "D", null),
+Row.ofKind(INSERT, "E", 4),
+Row.ofKind(INSERT, "E", 5),
+Row.ofKind(DELETE, "E", 5),
+Row.ofKind(UPDATE_BEFORE, "E", 4),
+Row.ofKind(UPDATE_AFTER, "E", 6)))
+.testResult(
+source ->
+"SELECT f0, array_agg(f1) FROM " +
source + " GROUP BY f0",
+TableApiAggSpec.groupBySelect(
+Collections.singletonList($("f0")),
+$("f0"),
+$("f1").arrayAgg()),
+ROW(STRING(), ARRAY(INT())),
+ROW(STRING(), ARRAY(INT())),
+Arrays.asList(
+Row.of("A", new Integer[] {1, 2}),
+Row.of("B", new Integer[] {2, 2, 3}),
+Row.of("C", new Integer[] {3}),
Review Comment:
It's ok to support it only partially however the main concern here:
~~I would expect more consistent result
since there is also input `Row.ofKind(INSERT, "D", null),` and expected
value `Row.of("D", null),`
So I would expect for ths either containing nulls or both not containing
null however not mixed~~
UPD: to be more clear:
after playing with Postgres and BigQuery I noticed that both respect nulls
by default and for BigQuery to make it ignoring nulls it should be specified
explicitely. Is there a reason why for Flink it is done differently? Probably
it's better to have similar behaviour
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
snuyanzin commented on code in PR #23411:
URL: https://github.com/apache/flink/pull/23411#discussion_r1452991614
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/ArrayAggFunctionITCase.java:
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.INT;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.types.RowKind.DELETE;
+import static org.apache.flink.types.RowKind.INSERT;
+import static org.apache.flink.types.RowKind.UPDATE_AFTER;
+import static org.apache.flink.types.RowKind.UPDATE_BEFORE;
+
+/** Tests for built-in ARRAY_AGG aggregation functions. */
+class ArrayAggFunctionITCase extends BuiltInAggregateFunctionTestBase {
+
+@Override
+Stream getTestCaseSpecs() {
+return Stream.of(
+TestSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_AGG)
+.withDescription("ARRAY changelog stream aggregation")
+.withSource(
+ROW(STRING(), INT()),
+Arrays.asList(
+Row.ofKind(INSERT, "A", 1),
+Row.ofKind(INSERT, "A", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 3),
+Row.ofKind(INSERT, "C", 3),
+Row.ofKind(INSERT, "C", null),
+Row.ofKind(INSERT, "D", null),
+Row.ofKind(INSERT, "E", 4),
+Row.ofKind(INSERT, "E", 5),
+Row.ofKind(DELETE, "E", 5),
+Row.ofKind(UPDATE_BEFORE, "E", 4),
+Row.ofKind(UPDATE_AFTER, "E", 6)))
+.testResult(
+source ->
+"SELECT f0, array_agg(f1) FROM " +
source + " GROUP BY f0",
+TableApiAggSpec.groupBySelect(
+Collections.singletonList($("f0")),
+$("f0"),
+$("f1").arrayAgg()),
+ROW(STRING(), ARRAY(INT())),
+ROW(STRING(), ARRAY(INT())),
+Arrays.asList(
+Row.of("A", new Integer[] {1, 2}),
+Row.of("B", new Integer[] {2, 2, 3}),
+Row.of("C", new Integer[] {3}),
Review Comment:
It's ok to support it only partially however the main concern here:
~~I would expect more consistent result
since there is also input `Row.ofKind(INSERT, "D", null),` and expected
value `Row.of("D", null),`
So I would expect for ths either containing nulls or both not containing
null however not mixed~~
UPD: to be more clear:
after playing with Postgres and BigQuery I noticed that both respect nulls
by default and for BigQuery to make it ignoring nulls it should be specified
explicitely. Is there a reason why for Flink it is done differently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please
Re: [PR] [FLINK-33450][autoscaler] Support the JDBCAutoScalerStateStore [flink-kubernetes-operator]
1996fanrui commented on code in PR #741:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/741#discussion_r1453006111
##
flink-autoscaler-plugin-jdbc/src/main/java/org/apache/flink/autoscaler/jdbc/state/JdbcAutoScalerStateStore.java:
##
@@ -0,0 +1,286 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.autoscaler.jdbc.state;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.autoscaler.JobAutoScalerContext;
+import org.apache.flink.autoscaler.ScalingSummary;
+import org.apache.flink.autoscaler.ScalingTracking;
+import org.apache.flink.autoscaler.metrics.CollectedMetrics;
+import org.apache.flink.autoscaler.state.AutoScalerStateStore;
+import org.apache.flink.autoscaler.utils.AutoScalerSerDeModule;
+import org.apache.flink.configuration.ConfigurationUtils;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+
+import com.fasterxml.jackson.core.JacksonException;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.datatype.jsr310.JavaTimeModule;
+import org.apache.commons.io.IOUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+
+import java.io.ByteArrayInputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.util.Base64;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+import java.util.SortedMap;
+import java.util.TreeMap;
+import java.util.zip.GZIPInputStream;
+import java.util.zip.GZIPOutputStream;
+
+import static
org.apache.flink.autoscaler.jdbc.state.StateType.COLLECTED_METRICS;
+import static
org.apache.flink.autoscaler.jdbc.state.StateType.PARALLELISM_OVERRIDES;
+import static org.apache.flink.autoscaler.jdbc.state.StateType.SCALING_HISTORY;
+import static
org.apache.flink.autoscaler.jdbc.state.StateType.SCALING_TRACKING;
+
+/**
+ * The state store which persists its state in JDBC related database.
+ *
+ * @param The job key.
+ * @param The job autoscaler context.
+ */
+@Experimental
+public class JdbcAutoScalerStateStore>
+implements AutoScalerStateStore {
+
+private static final Logger LOG =
LoggerFactory.getLogger(JdbcAutoScalerStateStore.class);
+
+private final JdbcStateStore jdbcStateStore;
+
+protected static final ObjectMapper YAML_MAPPER =
+new ObjectMapper()
+.registerModule(new JavaTimeModule())
+.registerModule(new AutoScalerSerDeModule());
+
+public JdbcAutoScalerStateStore(JdbcStateStore jdbcStateStore) {
+this.jdbcStateStore = jdbcStateStore;
+}
+
+@Override
+public void storeScalingHistory(
+Context jobContext, Map> scalingHistory)
+throws Exception {
+jdbcStateStore.putSerializedState(
+getSerializeKey(jobContext),
+SCALING_HISTORY,
+serializeScalingHistory(scalingHistory));
+}
+
+@Nonnull
+@Override
+public Map>
getScalingHistory(
+Context jobContext) {
+Optional serializedScalingHistory =
+jdbcStateStore.getSerializedState(getSerializeKey(jobContext),
SCALING_HISTORY);
+if (serializedScalingHistory.isEmpty()) {
+return new HashMap<>();
+}
+try {
+return deserializeScalingHistory(serializedScalingHistory.get());
+} catch (JacksonException e) {
+LOG.error(
+"Could not deserialize scaling history, possibly the
format changed. Discarding...",
+e);
+jdbcStateStore.removeSerializedState(getSerializeKey(jobContext),
SCALING_HISTORY);
+return new HashMap<>();
+}
+}
+
+@Override
+public void removeScalingHistory(Context jobContext) {
+jdbcStateStore.removeSerializedState(getSerializeKey(jobContext),
SCALING_HISTORY);
+}
+
+@Override
+public void storeScalingTracking(Context jobContext, ScalingTracking
scalingTrack)
+
Re: [PR] [FLINK-33450][autoscaler] Support the JDBCAutoScalerStateStore [flink-kubernetes-operator]
1996fanrui commented on code in PR #741:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/741#discussion_r1453004963
##
docs/content/docs/custom-resource/autoscaler.md:
##
@@ -260,17 +260,46 @@ job.autoscaler.metrics.window : 3m
> `ScalingReport` will show the recommended parallelism for each vertex.
After the flink job starts, please start the StandaloneAutoscaler process by
the
-following command.
+following command. Please download released autoscaler-standalone jar from
+[here](https://repo.maven.apache.org/maven2/org/apache/flink/flink-autoscaler-standalone/)
first.
```
java -cp flink-autoscaler-standalone-{{< version >}}.jar \
org.apache.flink.autoscaler.standalone.StandaloneAutoscalerEntrypoint \
---flinkClusterHost localhost \
---flinkClusterPort 8081
+--autoscaler.standalone.fetcher.flink-cluster.host localhost \
+--autoscaler.standalone.fetcher.flink-cluster.port 8081
```
-Updating the `flinkClusterHost` and `flinkClusterPort` based on your flink
cluster.
-In general, the host and port are the same as Flink WebUI.
+Updating the `autoscaler.standalone.fetcher.flink-cluster.host` and
`autoscaler.standalone.fetcher.flink-cluster.port`
+based on your flink cluster. In general, the host and port are the same as
Flink WebUI.
+
+### Using the JDBC Autoscaler State Store
+
+A driver dependency is required to connect to a specified database. Here are
drivers currently supported,
+please download JDBC driver and initialize database and table first.
+
+| Driver | Group Id | Artifact Id| JAR
| Schema
|
+|:---|:---|:---|:|-|
+| MySQL | `mysql`| `mysql-connector-java` |
[Download](https://repo.maven.apache.org/maven2/mysql/mysql-connector-java/)
| [Table
DDL](https://github.com/apache/flink-kubernetes-operator/blob/main/flink-autoscaler-plugin-jdbc/src/main/resources/schema/mysql_schema.sql)
|
+| PostgreSQL | `org.postgresql` | `postgresql` |
[Download](https://jdbc.postgresql.org/download/)
| [Table
DDL](https://github.com/apache/flink-kubernetes-operator/blob/main/flink-autoscaler-plugin-jdbc/src/main/resources/schema/postgres_schema.sql)
|
+| Derby | `org.apache.derby` | `derby`|
[Download](http://db.apache.org/derby/derby_downloads.html)
| [Table
DDL](https://github.com/apache/flink-kubernetes-operator/blob/main/flink-autoscaler-plugin-jdbc/src/main/resources/schema/derby_schema.sql)
|
+
+```
+JDBC_DRIVER_JAR=./mysql-connector-java-8.0.30.jar
+# export the password of jdbc state store
+export STATE_STORE_JDBC_PWD=123456
+
+java -cp flink-autoscaler-standalone-{{< version >}}.jar:${JDBC_DRIVER_JAR} \
+org.apache.flink.autoscaler.standalone.StandaloneAutoscalerEntrypoint \
+--autoscaler.standalone.fetcher.flink-cluster.host localhost \
+--autoscaler.standalone.fetcher.flink-cluster.port 8081 \
+--autoscaler.standalone.state-store.type jdbc \
+--autoscaler.standalone.state-store.jdbc.url
jdbc:mysql://localhost:3306/flink_autoscaler \
+--autoscaler.standalone.state-store.jdbc.username root
Review Comment:
In general, the environment variable name doesn't need to be changed. Users
need to export the password using this environment variable.
So I didn't mention `password-env-variable` here. But in the beginning of
this doc, it mentioned how to export password. WDYT?
```
# export the password of jdbc state store
export STATE_STORE_JDBC_PWD=123456
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-33980) Reorganize job configuration
[ https://issues.apache.org/jira/browse/FLINK-33980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu closed FLINK-33980. --- Fix Version/s: 1.19.0 Resolution: Done master/release-1.19: 06b37089f0c1cdf70ca21970a40d15c3eaba07ed 290b633c4265540d481ac563454c7f4a3e706b9d 65b8b3baad6a27e6253a93701508ba24dc8fcfe0 d02ef1cebf302c56a0d9d51664d2c7fb6f5be932 eb8af0c589ce46b091f403e848c7dc84b7e3ee8b > Reorganize job configuration > > > Key: FLINK-33980 > URL: https://issues.apache.org/jira/browse/FLINK-33980 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Currently, job configuration in FLINK is spread out across different > components, including StreamExecutionEnvironment, CheckpointConfig, and > ExecutionConfig. This distribution leads to inconsistencies among the > configurations stored within these components. Furthermore, the methods used > to configure these components vary; some rely on complex Java objects, while > others use ConfigOption, which is a key-value configuration approach. This > variation complicates the effective management of job configurations. > Additionally, passing complex Java objects (e.g., StateBackend and > CheckpointStorage) between the environment, StreamGraph, and JobGraph adds > complexity to development. > With the completion of FLIP-381, it is now time to standardize and unify job > configuration in FLINK. The goals of this JIRA are as follows: > # Migrate configuration from non-ConfigOption objects to use ConfigOption. > # Adopt a single Configuration object to house all configurations. > # Create complex Java objects, such as RestartBackoffTimeStrategyFactory, > CheckpointStorage, and StateBackend, directly from the configuration on the > JM side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33980][core] Reorganize job configuration [flink]
zhuzhurk closed pull request #24025: [FLINK-33980][core] Reorganize job configuration URL: https://github.com/apache/flink/pull/24025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
snuyanzin commented on code in PR #23411:
URL: https://github.com/apache/flink/pull/23411#discussion_r1452991614
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/ArrayAggFunctionITCase.java:
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.INT;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.types.RowKind.DELETE;
+import static org.apache.flink.types.RowKind.INSERT;
+import static org.apache.flink.types.RowKind.UPDATE_AFTER;
+import static org.apache.flink.types.RowKind.UPDATE_BEFORE;
+
+/** Tests for built-in ARRAY_AGG aggregation functions. */
+class ArrayAggFunctionITCase extends BuiltInAggregateFunctionTestBase {
+
+@Override
+Stream getTestCaseSpecs() {
+return Stream.of(
+TestSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_AGG)
+.withDescription("ARRAY changelog stream aggregation")
+.withSource(
+ROW(STRING(), INT()),
+Arrays.asList(
+Row.ofKind(INSERT, "A", 1),
+Row.ofKind(INSERT, "A", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 3),
+Row.ofKind(INSERT, "C", 3),
+Row.ofKind(INSERT, "C", null),
+Row.ofKind(INSERT, "D", null),
+Row.ofKind(INSERT, "E", 4),
+Row.ofKind(INSERT, "E", 5),
+Row.ofKind(DELETE, "E", 5),
+Row.ofKind(UPDATE_BEFORE, "E", 4),
+Row.ofKind(UPDATE_AFTER, "E", 6)))
+.testResult(
+source ->
+"SELECT f0, array_agg(f1) FROM " +
source + " GROUP BY f0",
+TableApiAggSpec.groupBySelect(
+Collections.singletonList($("f0")),
+$("f0"),
+$("f1").arrayAgg()),
+ROW(STRING(), ARRAY(INT())),
+ROW(STRING(), ARRAY(INT())),
+Arrays.asList(
+Row.of("A", new Integer[] {1, 2}),
+Row.of("B", new Integer[] {2, 2, 3}),
+Row.of("C", new Integer[] {3}),
Review Comment:
It's ok to support it only partially however the main concern here: I would
expect more consistent result
since there is also input `Row.ofKind(INSERT, "D", null),` and expected
value `Row.of("D", null),`
So I would expect for ths either containing nulls or both not containing
null however not mixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Resolved] (FLINK-33712) FLIP-391: Deprecate RuntimeContext#getExecutionConfig
[ https://issues.apache.org/jira/browse/FLINK-33712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junrui Li resolved FLINK-33712. --- Fix Version/s: 1.19.0 Release Note: RuntimeContext#getExecutionConfig is now be deprecated in FLINK-1.19. And this method is planned to be removed in Flink-2.0. More details can be found at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465937. Resolution: Fixed > FLIP-391: Deprecate RuntimeContext#getExecutionConfig > - > > Key: FLINK-33712 > URL: https://issues.apache.org/jira/browse/FLINK-33712 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Reporter: Junrui Li >Assignee: Junrui Li >Priority: Major > Fix For: 1.19.0 > > > Deprecate the RuntimeContext#getExecutionConfig and introduce alternative > getter methods that allow users to access specific information without > exposing unnecessary runtime details. More details see: > [FLIP-391|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465937] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34106) user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider
[ https://issues.apache.org/jira/browse/FLINK-34106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807065#comment-17807065 ] Junning Liang commented on FLINK-34106: --- I have fixed the issue, please assign it to me~ > user defined source parallelism in ddl can't work if connector > ScanRuntimeProvider is SourceFunctionProvider > > > Key: FLINK-34106 > URL: https://issues.apache.org/jira/browse/FLINK-34106 > Project: Flink > Issue Type: Bug >Reporter: Junning Liang >Priority: Critical > > I implemented a connector using the SourceFunctionProvider myself and found > that setting scan parallelism is not effective. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34106) user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider
Junning Liang created FLINK-34106: - Summary: user defined source parallelism in ddl can't work if connector ScanRuntimeProvider is SourceFunctionProvider Key: FLINK-34106 URL: https://issues.apache.org/jira/browse/FLINK-34106 Project: Flink Issue Type: Bug Reporter: Junning Liang I implemented a connector using the SourceFunctionProvider myself and found that setting scan parallelism is not effective. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33803] Set observedGeneration at end of reconciliation [flink-kubernetes-operator]
gyfora commented on code in PR #755:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/755#discussion_r1452979030
##
flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java:
##
@@ -55,4 +55,7 @@ public class FlinkDeploymentStatus extends
CommonStatus {
/** Information about the TaskManagers for the scale subresource. */
private TaskManagerInfo taskManager;
+
+/** Last observed generation of the FlinkDeployment. */
+private Long observedGeneration;
Review Comment:
@justin-chen I would prefer to replace the implementation and call it out in
the release notes instead. That way we end up with a cleaner codebase.
Any concerns from your end @mxm ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
Jiabao-Sun commented on PR #23411: URL: https://github.com/apache/flink/pull/23411#issuecomment-1893141857 Thanks @snuyanzin @dawidwys for the review. Could you help review it again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
Jiabao-Sun commented on code in PR #23411:
URL: https://github.com/apache/flink/pull/23411#discussion_r1452975551
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/ArrayAggFunctionITCase.java:
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions;
+
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.stream.Stream;
+
+import static org.apache.flink.table.api.DataTypes.ARRAY;
+import static org.apache.flink.table.api.DataTypes.INT;
+import static org.apache.flink.table.api.DataTypes.ROW;
+import static org.apache.flink.table.api.DataTypes.STRING;
+import static org.apache.flink.table.api.Expressions.$;
+import static org.apache.flink.types.RowKind.DELETE;
+import static org.apache.flink.types.RowKind.INSERT;
+import static org.apache.flink.types.RowKind.UPDATE_AFTER;
+import static org.apache.flink.types.RowKind.UPDATE_BEFORE;
+
+/** Tests for built-in ARRAY_AGG aggregation functions. */
+class ArrayAggFunctionITCase extends BuiltInAggregateFunctionTestBase {
+
+@Override
+Stream getTestCaseSpecs() {
+return Stream.of(
+TestSpec.forFunction(BuiltInFunctionDefinitions.ARRAY_AGG)
+.withDescription("ARRAY changelog stream aggregation")
+.withSource(
+ROW(STRING(), INT()),
+Arrays.asList(
+Row.ofKind(INSERT, "A", 1),
+Row.ofKind(INSERT, "A", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 2),
+Row.ofKind(INSERT, "B", 3),
+Row.ofKind(INSERT, "C", 3),
+Row.ofKind(INSERT, "C", null),
+Row.ofKind(INSERT, "D", null),
+Row.ofKind(INSERT, "E", 4),
+Row.ofKind(INSERT, "E", 5),
+Row.ofKind(DELETE, "E", 5),
+Row.ofKind(UPDATE_BEFORE, "E", 4),
+Row.ofKind(UPDATE_AFTER, "E", 6)))
+.testResult(
+source ->
+"SELECT f0, array_agg(f1) FROM " +
source + " GROUP BY f0",
+TableApiAggSpec.groupBySelect(
+Collections.singletonList($("f0")),
+$("f0"),
+$("f1").arrayAgg()),
+ROW(STRING(), ARRAY(INT())),
+ROW(STRING(), ARRAY(INT())),
+Arrays.asList(
+Row.of("A", new Integer[] {1, 2}),
+Row.of("B", new Integer[] {2, 2, 3}),
+Row.of("C", new Integer[] {3}),
Review Comment:
```sql
-- calcite
ARRAY_AGG([ ALL | DISTINCT ] value [ RESPECT NULLS | IGNORE NULLS ] [ ORDER
BY orderItem [, orderItem ]* ] )
-- flink
ARRAY_AGG([ ALL | DISTINCT ] expression)
```
This function simplifies some aspects compared to calcite.
1. Currently, the Flink parser does not support parsing RESPECT NULLS |
IGNORE NULLS, but it is still possible to make changes to support it.
2. AggregateUtil#extractDistinctInformation 848~858 ignores the ignoreNulls
fields.
https://github.com/apache/flink/blob/6bdb4f752adb2b43dbadd8ad4fffcb4c00568dd3/flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/AggregateUtil.scala#L848-L858
3. ListView does not support null values.
Re: [PR] [FLINK-33634] Add Conditions to Flink CRD's Status field [flink-kubernetes-operator]
lajith2006 commented on code in PR #749:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/749#discussion_r1452972476
##
flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/ConditionUtils.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.operator.api.utils;
+
+import io.fabric8.kubernetes.api.model.Condition;
+import io.fabric8.kubernetes.api.model.ConditionBuilder;
+
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/** Status of CR. */
+public class ConditionUtils {
+
+public static Condition ready(final String message) {
+return crCondition("Ready", "True", message, "Ready");
+}
+
+public static Condition notReady(final String message) {
+return crCondition("Ready", "False", message, "Progressing");
+}
+
+public static Condition error(final String message) {
+return crCondition("Error", "True", message, "The job terminally
failed");
+}
+
+public static Condition crCondition(
+final String type, final String status, final String message,
final String reason) {
+return new ConditionBuilder()
+.withType(type)
+.withStatus(status)
+.withMessage(message)
+.withReason(reason)
Review Comment:
As per the doc , https://maelvls.dev/kubernetes-conditions/ , looks like we
can keep reason as simple category of cause of the current status.
`Reason is intended to be used in concise output, such as one-line kubectl
get output, and in summarizing occurrences of causes, whereas Message is
intended to be presented to users in detailed status explanations, such as
kubectl describe output.`
I was thinking probably we can keep that for any not ready and error
conditions, and not required for ready condition?.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34088) fix the problems with special table name characters of postgres and oracle and sqlserver.
[ https://issues.apache.org/jira/browse/FLINK-34088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807062#comment-17807062 ] blackpighe commented on FLINK-34088: https://github.com/apache/flink-connector-jdbc/pull/89 > fix the problems with special table name characters of postgres and oracle > and sqlserver. > -- > > Key: FLINK-34088 > URL: https://issues.apache.org/jira/browse/FLINK-34088 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: jdbc-3.1.1 >Reporter: blackpighe >Priority: Major > Fix For: jdbc-3.1.1 > > Attachments: image-2024-01-15-17-10-57-586.png, > image-2024-01-15-17-11-11-136.png, image-2024-01-15-17-13-16-448.png > > Original Estimate: 48h > Remaining Estimate: 48h > > > !image-2024-01-15-17-11-11-136.png! > [https://github.com/apache/flink-connector-jdbc/] > The new version of flink jdbc is stand-alone and does not have an open issue > entry. > > The FLink-JDBC-connector does not handle special table names for oracle and > postgres and sqlserver, when the table name is a special character such as > default. jdbc will report an error. hopefully giving me a chance to fix the > problem, I can provide a commit > > > !image-2024-01-15-17-13-16-448.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. [flink-connector-jdbc]
boring-cyborg[bot] commented on PR #89: URL: https://github.com/apache/flink-connector-jdbc/pull/89#issuecomment-1893138077 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. [flink-connector-jdbc]
BlackPigHe opened a new pull request, #89: URL: https://github.com/apache/flink-connector-jdbc/pull/89 [hotfix] fix the problems with special table name characters of postgres and oracle and sqlserver. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33634] Add Conditions to Flink CRD's Status field [flink-kubernetes-operator]
lajith2006 commented on code in PR #749:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/749#discussion_r1452972476
##
flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/utils/ConditionUtils.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.operator.api.utils;
+
+import io.fabric8.kubernetes.api.model.Condition;
+import io.fabric8.kubernetes.api.model.ConditionBuilder;
+
+import java.text.SimpleDateFormat;
+import java.util.Date;
+
+/** Status of CR. */
+public class ConditionUtils {
+
+public static Condition ready(final String message) {
+return crCondition("Ready", "True", message, "Ready");
+}
+
+public static Condition notReady(final String message) {
+return crCondition("Ready", "False", message, "Progressing");
+}
+
+public static Condition error(final String message) {
+return crCondition("Error", "True", message, "The job terminally
failed");
+}
+
+public static Condition crCondition(
+final String type, final String status, final String message,
final String reason) {
+return new ConditionBuilder()
+.withType(type)
+.withStatus(status)
+.withMessage(message)
+.withReason(reason)
Review Comment:
As per the doc , https://maelvls.dev/kubernetes-conditions/ , looks like we
can keep reason as simple category of cause of the current status.
`Reason is intended to be used in concise output, such as one-line kubectl
get output, and in summarizing occurrences of causes, whereas Message is
intended to be presented to users in detailed status explanations, such as
kubectl describe output.`
I was thinking we can keep that for any not ready and error conditions, and
not required for ready conditions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33634] Add Conditions to Flink CRD's Status field [flink-kubernetes-operator]
gyfora commented on code in PR #749:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/749#discussion_r1452970570
##
flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/status/CommonStatus.java:
##
@@ -101,4 +108,67 @@ public ResourceLifecycleState getLifecycleState() {
* loop immediately. For example autoscaler overrides have changed and we
need to apply them.
*/
@JsonIgnore @Internal private boolean immediateReconciliationNeeded =
false;
+
+public List getConditions() {
+switch (getLifecycleState()) {
+case CREATED:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.notReady(
+"The resource was created in Kubernetes but
not yet handled by the operator"));
+break;
+case SUSPENDED:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.notReady("The resource (job) has been
suspended"));
+break;
+case UPGRADING:
+updateConditionIfNotExist(
+conditions, ConditionUtils.notReady("The resource is
being upgraded"));
+break;
+case DEPLOYED:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.ready(
+"The resource is deployed, but it’s not yet
considered to be stable and might be rolled back in the future"));
+break;
+case ROLLING_BACK:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.notReady(
+"The resource is being rolled back to the last
stable spec"));
+break;
+case ROLLED_BACK:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.ready("The resource is deployed with
the last stable spec"));
+break;
+case FAILED:
+updateConditionIfNotExist(conditions,
ConditionUtils.error("failed"));
+break;
+case STABLE:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.ready(
+"The resource deployment is considered to be
stable and won’t be rolled back"));
+break;
+}
+
+return conditions;
+}
+
+private void updateConditionIfNotExist(List conditions,
Condition newCondition) {
+if (conditions.isEmpty()) {
+conditions.add(newCondition);
+}
+if (conditions.stream()
+.noneMatch(condition ->
condition.getType().equals(newCondition.getType( {
+conditions.add(newCondition);
+} else if (conditions.removeIf(
+condition ->
+
!(condition.getReason().equals(newCondition.getReason())
+&&
condition.getMessage().equals(newCondition.getMessage() {
+conditions.add(newCondition);
+}
Review Comment:
As I wrote in the last comment, I think we need a FLIP for this instead of
fixing up this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33973) Add new interfaces for SinkV2 to synchronize the API with the SourceV2 API
[ https://issues.apache.org/jira/browse/FLINK-33973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17805957#comment-17805957 ] Gyula Fora edited comment on FLINK-33973 at 1/16/24 6:18 AM: - Merged to master 40cbf231e3dc15129b69136941adc15beebe8ef6 was (Author: gyfora): Merged to master 6afe98daf6190a77a67a1fa8ac8f12337a75f8e7 > Add new interfaces for SinkV2 to synchronize the API with the SourceV2 API > -- > > Key: FLINK-33973 > URL: https://issues.apache.org/jira/browse/FLINK-33973 > Project: Flink > Issue Type: Sub-task >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > > Create the new interfaces, set inheritance and deprecation to finalize the > interface. > After this change the new interafaces will exits, but they will not be > functional. > The existing interfaces, and test should be working without issue, to verify > that adding the API will be backward compatible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33973] Add new interfaces for SinkV2 to synchronize the API with the SourceV2 API [flink]
gyfora merged PR #24080: URL: https://github.com/apache/flink/pull/24080 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33634] Add Conditions to Flink CRD's Status field [flink-kubernetes-operator]
lajith2006 commented on code in PR #749:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/749#discussion_r1452966020
##
flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/status/CommonStatus.java:
##
@@ -101,4 +108,67 @@ public ResourceLifecycleState getLifecycleState() {
* loop immediately. For example autoscaler overrides have changed and we
need to apply them.
*/
@JsonIgnore @Internal private boolean immediateReconciliationNeeded =
false;
+
+public List getConditions() {
+switch (getLifecycleState()) {
+case CREATED:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.notReady(
+"The resource was created in Kubernetes but
not yet handled by the operator"));
+break;
+case SUSPENDED:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.notReady("The resource (job) has been
suspended"));
+break;
+case UPGRADING:
+updateConditionIfNotExist(
+conditions, ConditionUtils.notReady("The resource is
being upgraded"));
+break;
+case DEPLOYED:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.ready(
+"The resource is deployed, but it’s not yet
considered to be stable and might be rolled back in the future"));
+break;
+case ROLLING_BACK:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.notReady(
+"The resource is being rolled back to the last
stable spec"));
+break;
+case ROLLED_BACK:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.ready("The resource is deployed with
the last stable spec"));
+break;
+case FAILED:
+updateConditionIfNotExist(conditions,
ConditionUtils.error("failed"));
+break;
+case STABLE:
+updateConditionIfNotExist(
+conditions,
+ConditionUtils.ready(
+"The resource deployment is considered to be
stable and won’t be rolled back"));
+break;
+}
+
+return conditions;
+}
+
+private void updateConditionIfNotExist(List conditions,
Condition newCondition) {
+if (conditions.isEmpty()) {
+conditions.add(newCondition);
+}
+if (conditions.stream()
+.noneMatch(condition ->
condition.getType().equals(newCondition.getType( {
+conditions.add(newCondition);
+} else if (conditions.removeIf(
+condition ->
+
!(condition.getReason().equals(newCondition.getReason())
+&&
condition.getMessage().equals(newCondition.getMessage() {
+conditions.add(newCondition);
+}
Review Comment:
Sure, will add the docs to code. above logic is to make sure that rather to
blindly replace any existing conditions with new one , check for existing
condition with same type and replace only if the same condition type has
different message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks
[ https://issues.apache.org/jira/browse/FLINK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-34105: Priority: Critical (was: Major) > Akka timeout happens in TPC-DS benchmarks > - > > Key: FLINK-34105 > URL: https://issues.apache.org/jira/browse/FLINK-34105 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: Zhu Zhu >Priority: Critical > Attachments: image-2024-01-16-13-59-45-556.png > > > We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The > problem did not happen in 1.18.0. > After bisecting, we find the problem was introduced in FLINK-33532. > !image-2024-01-16-13-59-45-556.png|width=800! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks
[ https://issues.apache.org/jira/browse/FLINK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807052#comment-17807052 ] Zhu Zhu edited comment on FLINK-34105 at 1/16/24 6:06 AM: -- [~lsdy] [~guoyangze] would you help to take a look? This indicates a problem that existing large Flink batch jobs may become unstable in 1.19. was (Author: zhuzh): [~lsdy] [~guoyangze] would you help to take a look? > Akka timeout happens in TPC-DS benchmarks > - > > Key: FLINK-34105 > URL: https://issues.apache.org/jira/browse/FLINK-34105 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: Zhu Zhu >Priority: Major > Attachments: image-2024-01-16-13-59-45-556.png > > > We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The > problem did not happen in 1.18.0. > After bisecting, we find the problem was introduced in FLINK-33532. > !image-2024-01-16-13-59-45-556.png|width=800! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks
[ https://issues.apache.org/jira/browse/FLINK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-34105: Description: We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The problem did not happen in 1.18.0. After bisecting, we find the problem was introduced in FLINK-33532. !image-2024-01-16-13-59-45-556.png|width=250! was: We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The problem did not happen in 1.18.0. After bisecting, we find the problem was introduced in FLINK-33532. !image-2024-01-16-13-59-45-556.png! > Akka timeout happens in TPC-DS benchmarks > - > > Key: FLINK-34105 > URL: https://issues.apache.org/jira/browse/FLINK-34105 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: Zhu Zhu >Priority: Major > Attachments: image-2024-01-16-13-59-45-556.png > > > We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The > problem did not happen in 1.18.0. > After bisecting, we find the problem was introduced in FLINK-33532. > !image-2024-01-16-13-59-45-556.png|width=250! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks
[ https://issues.apache.org/jira/browse/FLINK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-34105: Description: We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The problem did not happen in 1.18.0. After bisecting, we find the problem was introduced in FLINK-33532. !image-2024-01-16-13-59-45-556.png|width=800! was: We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The problem did not happen in 1.18.0. After bisecting, we find the problem was introduced in FLINK-33532. !image-2024-01-16-13-59-45-556.png|width=600! > Akka timeout happens in TPC-DS benchmarks > - > > Key: FLINK-34105 > URL: https://issues.apache.org/jira/browse/FLINK-34105 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: Zhu Zhu >Priority: Major > Attachments: image-2024-01-16-13-59-45-556.png > > > We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The > problem did not happen in 1.18.0. > After bisecting, we find the problem was introduced in FLINK-33532. > !image-2024-01-16-13-59-45-556.png|width=800! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks
[ https://issues.apache.org/jira/browse/FLINK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-34105: Description: We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The problem did not happen in 1.18.0. After bisecting, we find the problem was introduced in FLINK-33532. !image-2024-01-16-13-59-45-556.png|width=600! was: We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The problem did not happen in 1.18.0. After bisecting, we find the problem was introduced in FLINK-33532. !image-2024-01-16-13-59-45-556.png|width=250! > Akka timeout happens in TPC-DS benchmarks > - > > Key: FLINK-34105 > URL: https://issues.apache.org/jira/browse/FLINK-34105 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: Zhu Zhu >Priority: Major > Attachments: image-2024-01-16-13-59-45-556.png > > > We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The > problem did not happen in 1.18.0. > After bisecting, we find the problem was introduced in FLINK-33532. > !image-2024-01-16-13-59-45-556.png|width=600! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34104][autoscaler] Improve the ScalingReport format of autoscaling [flink-kubernetes-operator]
1996fanrui opened a new pull request, #757: URL: https://github.com/apache/flink-kubernetes-operator/pull/757 ## What is the purpose of the change Currently, the scaling report format is `Vertex ID %s | Parallelism %d -> %d | Processing capacity %.2f -> %.2f | Target data rate %.2f`. It has 2 disadvantages: 1. When one job has multiple vertices, the report of all vertices are mixed together without any separator, here is an example: - `Scaling execution enabled, begin scaling vertices: Vertex ID ea632d67b7d595e5b851708ae9ad79d6 | Parallelism 2 -> 1 | Processing capacity 800466.67 -> 320186.67 | Target data rate 715.10 Vertex ID bc764cd8ddf7a0cff126f51c16239658 | Parallelism 2 -> 1 | Processing capacity 79252.08 -> 31700.83 | Target data rate 895.93 Vertex ID 0a448493b4782967b150582570326227 | Parallelism 8 -> 16 | Processing capacity 716.05 -> 1141.00 | Target data rate 715.54` - We can see the Vertex ID is the beginning of each vertex report, it doesn't have any separator with the last vertex. 2. This format is non-standard, it's hard to deserialize. - When job enables the autoscaler and disable the scaling. - Flink platform maintainer wants to show the scaling report in WebUI, it's helpful to using the report result for flink users. - So easy to deserialize is useful for these flink platform. ## Brief change log Serializing the scaling report with json format, it's easy to read and deserialize. Here is the scaling report message after this PR: `` ## Verifying this change - Added `AutoScalerEventHandlerTest` to test the ScalingReport format. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks
[ https://issues.apache.org/jira/browse/FLINK-34105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807052#comment-17807052 ] Zhu Zhu commented on FLINK-34105: - [~lsdy] [~guoyangze] would you help to take a look? > Akka timeout happens in TPC-DS benchmarks > - > > Key: FLINK-34105 > URL: https://issues.apache.org/jira/browse/FLINK-34105 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.19.0 >Reporter: Zhu Zhu >Priority: Major > Attachments: image-2024-01-16-13-59-45-556.png > > > We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The > problem did not happen in 1.18.0. > After bisecting, we find the problem was introduced in FLINK-33532. > !image-2024-01-16-13-59-45-556.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34104) Improve the ScalingReport format of autoscaling
[
https://issues.apache.org/jira/browse/FLINK-34104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-34104:
---
Labels: pull-request-available (was: )
> Improve the ScalingReport format of autoscaling
> ---
>
> Key: FLINK-34104
> URL: https://issues.apache.org/jira/browse/FLINK-34104
> Project: Flink
> Issue Type: Improvement
> Components: Autoscaler
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: kubernetes-operator-1.8.0
>
>
> Currently, the scaling report format is
> {color:#6a8759} Vertex ID %s | Parallelism %d -> %d | Processing capacity
> %.2f -> %.2f | Target data rate %.2f{color}
> {color:#172b4d}It has 2 disadvantages:{color}
> # {color:#172b4d}When one job has multiple vertices, the report of all
> vertices are mixed together without any separator{color}{color:#172b4d}, here
> is an example:{color}
> ** {color:#172b4d}Scaling execution enabled, begin scaling vertices: Vertex
> ID ea632d67b7d595e5b851708ae9ad79d6 | Parallelism 2 -> 1 | Processing
> capacity 800466.67 -> 320186.67 | Target data rate 715.10 Vertex ID
> bc764cd8ddf7a0cff126f51c16239658 | Parallelism 2 -> 1 | Processing capacity
> 79252.08 -> 31700.83 | Target data rate 895.93 Vertex ID
> 0a448493b4782967b150582570326227 | Parallelism 8 -> 16 | Processing capacity
> 716.05 -> 1141.00 | Target data rate 715.54{color}
> ** {color:#172b4d}We can see the Vertex ID is the beginning of each vertex
> report, it doesn't have any {color}{color:#172b4d}separator with the last
> vertex.
> {color}
> # {color:#172b4d}This format is non-standard{color}{color:#172b4d}, it's
> hard to deserialize.{color}
> ** {color:#172b4d}When job enables the autoscaler and disable the
> scaling.{color}
> ** {color:#172b4d}Flink platform maintainer wants to show the scaling report
> in WebUI, it's helpful to using the report result for flink users.{color}
> ** {color:#172b4d}So easy to deserialize is useful for these flink
> platform.{color}
> h2. {color:#172b4d}Solution:{color}
> {color:#172b4d}Serializing the scaling report with json format, it's easy to
> read and deserialize.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34105) Akka timeout happens in TPC-DS benchmarks
Zhu Zhu created FLINK-34105: --- Summary: Akka timeout happens in TPC-DS benchmarks Key: FLINK-34105 URL: https://issues.apache.org/jira/browse/FLINK-34105 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.19.0 Reporter: Zhu Zhu Attachments: image-2024-01-16-13-59-45-556.png We noticed akka timeout happens in 10TB TPC-DS benchmarks in 1.19. The problem did not happen in 1.18.0. After bisecting, we find the problem was introduced in FLINK-33532. !image-2024-01-16-13-59-45-556.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33973] Add new interfaces for SinkV2 to synchronize the API with the SourceV2 API [flink]
pvary commented on PR #24080: URL: https://github.com/apache/flink/pull/24080#issuecomment-1893114512 @gyfora: Please merge. We have a green test run. Thx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34104) Improve the ScalingReport format of autoscaling
Rui Fan created FLINK-34104:
---
Summary: Improve the ScalingReport format of autoscaling
Key: FLINK-34104
URL: https://issues.apache.org/jira/browse/FLINK-34104
Project: Flink
Issue Type: Improvement
Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
Fix For: kubernetes-operator-1.8.0
Currently, the scaling report format is
{color:#6a8759} Vertex ID %s | Parallelism %d -> %d | Processing capacity %.2f
-> %.2f | Target data rate %.2f{color}
{color:#172b4d}It has 2 disadvantages:{color}
# {color:#172b4d}When one job has multiple vertices, the report of all
vertices are mixed together without any separator{color}{color:#172b4d}, here
is an example:{color}
** {color:#172b4d}Scaling execution enabled, begin scaling vertices: Vertex ID
ea632d67b7d595e5b851708ae9ad79d6 | Parallelism 2 -> 1 | Processing capacity
800466.67 -> 320186.67 | Target data rate 715.10 Vertex ID
bc764cd8ddf7a0cff126f51c16239658 | Parallelism 2 -> 1 | Processing capacity
79252.08 -> 31700.83 | Target data rate 895.93 Vertex ID
0a448493b4782967b150582570326227 | Parallelism 8 -> 16 | Processing capacity
716.05 -> 1141.00 | Target data rate 715.54{color}
** {color:#172b4d}We can see the Vertex ID is the beginning of each vertex
report, it doesn't have any {color}{color:#172b4d}separator with the last
vertex.
{color}
# {color:#172b4d}This format is non-standard{color}{color:#172b4d}, it's hard
to deserialize.{color}
** {color:#172b4d}When job enables the autoscaler and disable the
scaling.{color}
** {color:#172b4d}Flink platform maintainer wants to show the scaling report
in WebUI, it's helpful to using the report result for flink users.{color}
** {color:#172b4d}So easy to deserialize is useful for these flink
platform.{color}
h2. {color:#172b4d}Solution:{color}
{color:#172b4d}Serializing the scaling report with json format, it's easy to
read and deserialize.{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34078][state]Move InternalKeyContext classes from o.a.f.runtim… [flink]
masteryhx closed pull request #24093: [FLINK-34078][state]Move InternalKeyContext classes from o.a.f.runtim… URL: https://github.com/apache/flink/pull/24093 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34103) AsyncIO example failed to run as DataGen Connector is not bundled
[ https://issues.apache.org/jira/browse/FLINK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanghao Chen updated FLINK-34103: -- External issue ID: (was: FLINK-32821) > AsyncIO example failed to run as DataGen Connector is not bundled > - > > Key: FLINK-34103 > URL: https://issues.apache.org/jira/browse/FLINK-34103 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Priority: Major > > From the comments of FLINK-32821: > root@73186f600374:/opt/flink# bin/flink run > /volume/flink-examples-streaming-1.18.0-AsyncIO.jar > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > java.lang.NoClassDefFoundError: > org/apache/flink/connector/datagen/source/DataGeneratorSource > at > org.apache.flink.streaming.examples.async.AsyncIOExample.main(AsyncIOExample.java:82) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown > Source) > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > at java.base/java.lang.reflect.Method.invoke(Unknown Source) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) > at > org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) > at > org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) > at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) > Caused by: java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.DataGeneratorSource > at java.base/java.net.URLClassLoader.findClass(Unknown Source) > at java.base/java.lang.ClassLoader.loadClass(Unknown Source) > at > org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67) > at > org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65) > at > org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) > at java.base/java.lang.ClassLoader.loadClass(Unknown Source) > ... 15 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34103) AsyncIO example failed to run as DataGen Connector is not bundled
[ https://issues.apache.org/jira/browse/FLINK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanghao Chen updated FLINK-34103: -- External issue ID: FLINK-32821 > AsyncIO example failed to run as DataGen Connector is not bundled > - > > Key: FLINK-34103 > URL: https://issues.apache.org/jira/browse/FLINK-34103 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Priority: Major > > From the comments of FLINK-32821: > root@73186f600374:/opt/flink# bin/flink run > /volume/flink-examples-streaming-1.18.0-AsyncIO.jar > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > WARNING: Unknown module: jdk.compiler specified to --add-exports > java.lang.NoClassDefFoundError: > org/apache/flink/connector/datagen/source/DataGeneratorSource > at > org.apache.flink.streaming.examples.async.AsyncIOExample.main(AsyncIOExample.java:82) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown > Source) > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > at java.base/java.lang.reflect.Method.invoke(Unknown Source) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > at > org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245) > at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) > at > org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) > at > org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) > at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) > Caused by: java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.DataGeneratorSource > at java.base/java.net.URLClassLoader.findClass(Unknown Source) > at java.base/java.lang.ClassLoader.loadClass(Unknown Source) > at > org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67) > at > org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65) > at > org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) > at java.base/java.lang.ClassLoader.loadClass(Unknown Source) > ... 15 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32821) Streaming examples failed to execute due to error in packaging
[ https://issues.apache.org/jira/browse/FLINK-32821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807047#comment-17807047 ] Zhanghao Chen commented on FLINK-32821: --- FYI [~qinjunjerry] , I've created [FLINK-34103] AsyncIO example failed to run as DataGen Connector is not bundled - ASF JIRA (apache.org) to track this issue. > Streaming examples failed to execute due to error in packaging > -- > > Key: FLINK-32821 > URL: https://issues.apache.org/jira/browse/FLINK-32821 > Project: Flink > Issue Type: Bug > Components: Examples >Affects Versions: 1.18.0 >Reporter: Zhanghao Chen >Assignee: Zhanghao Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > > 6 out of the 8 streaming examples failed to run: > * Iteration & SessionWindowing & SocketWindowWordCount & WindowJoin failed > to run due to java.lang.NoClassDefFoundError: > org/apache/flink/streaming/examples/utils/ParameterTool > * MatrixVectorMul & TopSpeedWindowing failed to run due to: Caused by: > java.lang.ClassNotFoundException: > org.apache.flink.connector.datagen.source.GeneratorFunction > The NoClassDefFoundError with ParameterTool is introduced by FLINK-32558 > Properly deprecate DataSet API - ASF JIRA (apache.org), and we'd better > resolve FLINK-32820 ParameterTool is mistakenly marked as deprecated - ASF > JIRA (apache.org) first before we come to a fix for this problem. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34103) AsyncIO example failed to run as DataGen Connector is not bundled
[ https://issues.apache.org/jira/browse/FLINK-34103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanghao Chen updated FLINK-34103: -- Description: >From the comments of FLINK-32821: root@73186f600374:/opt/flink# bin/flink run /volume/flink-examples-streaming-1.18.0-AsyncIO.jar WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports java.lang.NoClassDefFoundError: org/apache/flink/connector/datagen/source/DataGeneratorSource at org.apache.flink.streaming.examples.async.AsyncIOExample.main(AsyncIOExample.java:82) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) Caused by: java.lang.ClassNotFoundException: org.apache.flink.connector.datagen.source.DataGeneratorSource at java.base/java.net.URLClassLoader.findClass(Unknown Source) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67) at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ... 15 more was: >From the comments of >[FLINK-32821|https://issues.apache.org/jira/browse/FLINK-32821]: root@73186f600374:/opt/flink# bin/flink run /volume/flink-examples-streaming-1.18.0-AsyncIO.jar WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports java.lang.NoClassDefFoundError: org/apache/flink/connector/datagen/source/DataGeneratorSource at org.apache.flink.streaming.examples.async.AsyncIOExample.main(AsyncIOExample.java:82) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) Caused by: java.lang.ClassNotFoundException: org.apache.flink.connector.datagen.source.DataGeneratorSource at java.base/java.net.URLClassLoader.findClass(Unknown Source) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) at
[jira] [Created] (FLINK-34103) AsyncIO example failed to run as DataGen Connector is not bundled
Zhanghao Chen created FLINK-34103: - Summary: AsyncIO example failed to run as DataGen Connector is not bundled Key: FLINK-34103 URL: https://issues.apache.org/jira/browse/FLINK-34103 Project: Flink Issue Type: Bug Components: Examples Affects Versions: 1.18.0 Reporter: Zhanghao Chen >From the comments of >[FLINK-32821|https://issues.apache.org/jira/browse/FLINK-32821]: root@73186f600374:/opt/flink# bin/flink run /volume/flink-examples-streaming-1.18.0-AsyncIO.jar WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports WARNING: Unknown module: jdk.compiler specified to --add-exports java.lang.NoClassDefFoundError: org/apache/flink/connector/datagen/source/DataGeneratorSource at org.apache.flink.streaming.examples.async.AsyncIOExample.main(AsyncIOExample.java:82) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) Caused by: java.lang.ClassNotFoundException: org.apache.flink.connector.datagen.source.DataGeneratorSource at java.base/java.net.URLClassLoader.findClass(Unknown Source) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67) at org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:65) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51) at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ... 15 more -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34052) Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository
[ https://issues.apache.org/jira/browse/FLINK-34052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807045#comment-17807045 ] Zhanghao Chen commented on FLINK-34052: --- Thanks for reporting this, I'll take a look. [~huweihua] could you assign this to me? > Missing TopSpeedWindowing and SessionWindowing JARs in Flink Maven Repository > - > > Key: FLINK-34052 > URL: https://issues.apache.org/jira/browse/FLINK-34052 > Project: Flink > Issue Type: Bug > Components: Build System, Examples >Affects Versions: 1.18.0 >Reporter: Junrui Li >Priority: Major > > As a result of the changes implemented in FLINK-32821, the build process no > longer produces artifacts with the names > flink-examples-streaming-1.x-TopSpeedWindowing.jar and > flink-examples-streaming-1.x-SessionWindowing.jar. This has led to the > absence of these specific JAR files in the Maven repository > (https://repo1.maven.org/maven2/org/apache/flink/flink-examples-streaming/1.18.0/). > These artifacts were previously available and may still be expected by users > as part of their application dependencies. Their removal could potentially > break existing build pipelines and applications that depend on these example > JARs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33221][core][config] Add config options for administrator JVM options [flink]
X-czh commented on PR #24098: URL: https://github.com/apache/flink/pull/24098#issuecomment-1893096096 Hi @huwh @1996fanrui, do you have time to help review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33803] Set observedGeneration at end of reconciliation [flink-kubernetes-operator]
justin-chen commented on code in PR #755:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/755#discussion_r1452942179
##
flink-kubernetes-operator-api/src/main/java/org/apache/flink/kubernetes/operator/api/status/FlinkDeploymentStatus.java:
##
@@ -55,4 +55,7 @@ public class FlinkDeploymentStatus extends
CommonStatus {
/** Information about the TaskManagers for the scale subresource. */
private TaskManagerInfo taskManager;
+
+/** Last observed generation of the FlinkDeployment. */
+private Long observedGeneration;
Review Comment:
Thanks for the guidance - I've added `observedGeneration` to `CommonStatus`
class and set it alongside the `lastReconciledSpec`. I've left the `generation`
within the `lastReconciledSpec` intact in case there are existing client
dependencies on it. Let me know about any additional changes to be made
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-34102) Invalid configuration when using 'env.log.max' on yarn application mode
RocMarshal created FLINK-34102: -- Summary: Invalid configuration when using 'env.log.max' on yarn application mode Key: FLINK-34102 URL: https://issues.apache.org/jira/browse/FLINK-34102 Project: Flink Issue Type: Bug Components: Runtime / Configuration Affects Versions: 1.19.0 Reporter: RocMarshal -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33988][configuration] Fix the invalid configuration when using initialized root logger level on yarn deployment mode [flink]
1996fanrui commented on PR #24042: URL: https://github.com/apache/flink/pull/24042#issuecomment-1893069359 > I made a fix like the env-value passing-in at https://github.com/RocMarshal/flink/commit/624c199e756ed5f04f849e8e53033bc1ef258f93 > Would we merge the additional fix into the current pr ? > Please let me know what's your opinion~ :) Same PR is fine for me, but separate JIRA and commits are necessary. Separate JIRA and commits are easy to follow up and cherry pick for other users. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33768] Support dynamic source parallelism inference for batch jobs [flink]
SinBex commented on PR #24087: URL: https://github.com/apache/flink/pull/24087#issuecomment-1893062734 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25054) Improve exception message for unsupported hashLength for SHA2 function
[ https://issues.apache.org/jira/browse/FLINK-25054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-25054: --- Labels: pull-request-available (was: ) > Improve exception message for unsupported hashLength for SHA2 function > -- > > Key: FLINK-25054 > URL: https://issues.apache.org/jira/browse/FLINK-25054 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.12.3 >Reporter: DingGeGe >Assignee: Shuai Xu >Priority: Major > Labels: pull-request-available > Attachments: image-2021-11-25-16-59-56-699.png > > Original Estimate: 1h > Remaining Estimate: 1h > > 【exception sql】 > SELECT > SHA2(, 128) > FROM > > 【effect】 > when sql is long , it`s hard to clear where is the problem on this issue > 【reason】 > build-in function SHA2, hashLength do not support “128”, but I could not > understand from > 【Exception log】 > !image-2021-11-25-16-59-56-699.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-25054][table-planner] Extend the exception message for SHA2 function [flink]
flinkbot commented on PR #24099: URL: https://github.com/apache/flink/pull/24099#issuecomment-1893037933 ## CI report: * 6525ba25342d12481b64e64fcc66ba6e57f0480f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33988][configuration] Fix the invalid configuration when using initialized root logger level on yarn deployment mode [flink]
RocMarshal commented on PR #24042: URL: https://github.com/apache/flink/pull/24042#issuecomment-1893035604 Thanks @X-czh for the double-check and @1996fanrui review. I made a fix like the env-value passing-in at https://github.com/RocMarshal/flink/commit/624c199e756ed5f04f849e8e53033bc1ef258f93 Would we merge the additional fix into the current pr ? Please let me know what's your opinion~ :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Extend the exception message for SHA2 function [flink]
xishuaidelin opened a new pull request, #24099: URL: https://github.com/apache/flink/pull/24099 ## What is the purpose of the change *This pull request extends the exception message for sha2 function.* ## Brief change log modify the function addReusableSha2MessageDigest. ## Verifying this change N/A ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no ) - The runtime per-record code paths (performance sensitive): (no ) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no ) - The S3 file system connector: (no ) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34087][tests][JUnit5 Migration] Migarate to junit5 of flink-dist module [flink]
Jiabao-Sun commented on PR #24092: URL: https://github.com/apache/flink/pull/24092#issuecomment-1893033205 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33856) Add metrics to monitor the interaction performance between task and external storage system in the process of checkpoint making
[ https://issues.apache.org/jira/browse/FLINK-33856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807019#comment-17807019 ] Jufang He edited comment on FLINK-33856 at 1/16/24 3:51 AM: ok, I'll get to work on it. [~pnowojski] was (Author: JIRAUSER302059): ok, I'll get to work on it. > Add metrics to monitor the interaction performance between task and external > storage system in the process of checkpoint making > --- > > Key: FLINK-33856 > URL: https://issues.apache.org/jira/browse/FLINK-33856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.18.0 >Reporter: Jufang He >Assignee: Jufang He >Priority: Major > Labels: pull-request-available > > When Flink makes a checkpoint, the interaction performance with the external > file system has a great impact on the overall time-consuming. Therefore, it > is easy to observe the bottleneck point by adding performance indicators when > the task interacts with the external file storage system. These include: the > rate of file write , the latency to write the file, the latency to close the > file. > In flink side add the above metrics has the following advantages: convenient > statistical different task E2E time-consuming; do not need to distinguish the > type of external storage system, can be unified in the > FsCheckpointStreamFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33856) Add metrics to monitor the interaction performance between task and external storage system in the process of checkpoint making
[ https://issues.apache.org/jira/browse/FLINK-33856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807019#comment-17807019 ] Jufang He commented on FLINK-33856: --- ok, I'll get to work on it. > Add metrics to monitor the interaction performance between task and external > storage system in the process of checkpoint making > --- > > Key: FLINK-33856 > URL: https://issues.apache.org/jira/browse/FLINK-33856 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.18.0 >Reporter: Jufang He >Assignee: Jufang He >Priority: Major > Labels: pull-request-available > > When Flink makes a checkpoint, the interaction performance with the external > file system has a great impact on the overall time-consuming. Therefore, it > is easy to observe the bottleneck point by adding performance indicators when > the task interacts with the external file storage system. These include: the > rate of file write , the latency to write the file, the latency to close the > file. > In flink side add the above metrics has the following advantages: convenient > statistical different task E2E time-consuming; do not need to distinguish the > type of external storage system, can be unified in the > FsCheckpointStreamFactory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-34007) Flink Job stuck in suspend state after losing leadership in HA Mode
[ https://issues.apache.org/jira/browse/FLINK-34007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807017#comment-17807017 ] Zhenqiu Huang edited comment on FLINK-34007 at 1/16/24 3:40 AM: [~mapohl] [~wangyang0918] I am intensively testing flink 1.18. Within two days, there are users reported the job manager stuck issue in 1.17 and 1.16. 1.18 and 1.17 job instances are running in the same cluster. 1.16 is in different cluster. I attached another LeaderElector-Debug.json file that contains debug log of a flink 1.18 app. The issue happened several times: 1. due to the configmap not accessible from api sever then renew timeout exceeded. 2. a failure on patch on a updated configmap The interesting part of the behavior of last several days is that job manager was not stuck but exit directly. Then, new job manager pod started correctly that is why new leader is selected in the log above. Hopefully, it is useful for your diagnosis. [~wangyang0918] >From my initial observation (before creating the jira), the leader annotation >update stopped when job manager was stuck. was (Author: zhenqiuhuang): [~mapohl] [~wangyang0918] I am intensively testing flink 1.18. Within two days, there are users reported the job manager stuck issue in 1.17 and 1.16. 1.18 and 1.17 job instances are running in the same cluster. 1.16 is in different cluster. I attached another LeaderElector-Debug.json file that contains debug log of a flink 1.18 app. The issue happened several times: 1. due to the configmap not accessible from api sever then renew timeout exceeded 2. a failure on patch on a updated configmap The interesting part of the behavior of last several days is that job manager was not stuck but exit directly. Then, new job manager pod started correctly that is why new leader is selected in the log above. Hopefully, it is useful for your diagnosis. > Flink Job stuck in suspend state after losing leadership in HA Mode > --- > > Key: FLINK-34007 > URL: https://issues.apache.org/jira/browse/FLINK-34007 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.16.3, 1.17.2, 1.18.1, 1.18.2 >Reporter: Zhenqiu Huang >Priority: Major > Attachments: Debug.log, LeaderElector-Debug.json, job-manager.log > > > The observation is that Job manager goes to suspend state with a failed > container not able to register itself to resource manager after timeout. > JM Log, see attached > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34101) use flink ui upload jar error, message is 'Exactly 1 file must be sent, received 0'
[ https://issues.apache.org/jira/browse/FLINK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807018#comment-17807018 ] blackpighe commented on FLINK-34101: !image-2024-01-16-11-37-26-343.png! this is copy curl request by web console： curl "http://localhost:8083/proxy/ingress/ui/1/224/jars/upload; ^ -H "Accept: application/json, text/plain, */*" ^ -H "Accept-Language: zh-CN,zh;q=0.9" ^ -H "Cache-Control: no-cache" ^ -H "Connection: keep-alive" ^ -H "Content-Type: multipart/form-data; boundary=WebKitFormBoundaryl6QGLTdn1N5xv5zc" ^ -H "Origin: http://localhost:8083; ^ -H "Pragma: no-cache" ^ -H "Referer: http://localhost:8083/proxy/ingress/ui/1/224/; ^ -H "Sec-Fetch-Dest: empty" ^ -H "Sec-Fetch-Mode: cors" ^ -H "Sec-Fetch-Site: same-origin" ^ -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" ^ -H "sec-ch-ua: ^\^"Not_A Brand^\^";v=^\^"8^\^", ^\^"Chromium^\^";v=^\^"120^\^", ^\^"Google Chrome^\^";v=^\^"120^\^"" ^ -H "sec-ch-ua-mobile: ?0" ^ -H "sec-ch-ua-platform: ^\^"Windows^\^"" ^ --data-raw ^"--WebKitFormBoundaryl6QGLTdn1N5xv5zc^ Content-Disposition: form-data; name=^\^"jarfile^\^"; filename=^\^"faas-flink-1.14.3_2.12-20481024-SNAPSHOT-session.jar^\^"^ Content-Type: application/octet-stream^ ^ ^ --WebKitFormBoundaryl6QGLTdn1N5xv5zc--^ ^" ^ --compressed > use flink ui upload jar error, message is 'Exactly 1 file must be sent, > received 0' > --- > > Key: FLINK-34101 > URL: https://issues.apache.org/jira/browse/FLINK-34101 > Project: Flink > Issue Type: Bug > Components: Runtime / REST, Runtime / Web Frontend >Affects Versions: 1.17.2 >Reporter: blackpighe >Priority: Major > Fix For: 1.17.2 > > Attachments: image-2024-01-16-11-32-50-015.png, > image-2024-01-16-11-35-21-582.png, image-2024-01-16-11-37-26-343.png > > > use flink ui upload jar, Something went wrong. > > !image-2024-01-16-11-32-50-015.png! > message is : > Server Response:org.apache.flink.runtime.rest.handler.RestHandlerException: > Exactly 1 file must be sent, received 0. at > org.apache.flink.runtime.webmonitor.handlers.JarUploadHandler.handleRequest(JarUploadHandler.java:80) > at > org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) > at > org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196) > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) > at java.util.Optional.ifPresent(Optional.java:159) at > org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at >
[jira] [Updated] (FLINK-34101) use flink ui upload jar error, message is 'Exactly 1 file must be sent, received 0'
[ https://issues.apache.org/jira/browse/FLINK-34101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] blackpighe updated FLINK-34101: --- Attachment: image-2024-01-16-11-37-26-343.png > use flink ui upload jar error, message is 'Exactly 1 file must be sent, > received 0' > --- > > Key: FLINK-34101 > URL: https://issues.apache.org/jira/browse/FLINK-34101 > Project: Flink > Issue Type: Bug > Components: Runtime / REST, Runtime / Web Frontend >Affects Versions: 1.17.2 >Reporter: blackpighe >Priority: Major > Fix For: 1.17.2 > > Attachments: image-2024-01-16-11-32-50-015.png, > image-2024-01-16-11-35-21-582.png, image-2024-01-16-11-37-26-343.png > > > use flink ui upload jar, Something went wrong. > > !image-2024-01-16-11-32-50-015.png! > message is : > Server Response:org.apache.flink.runtime.rest.handler.RestHandlerException: > Exactly 1 file must be sent, received 0. at > org.apache.flink.runtime.webmonitor.handlers.JarUploadHandler.handleRequest(JarUploadHandler.java:80) > at > org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) > at > org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196) > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) > at java.util.Optional.ifPresent(Optional.java:159) at > org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:203) > at > org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69) > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) > at > org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) > at >
[jira] [Created] (FLINK-34101) use flink ui upload jar error, message is 'Exactly 1 file must be sent, received 0'
blackpighe created FLINK-34101: -- Summary: use flink ui upload jar error, message is 'Exactly 1 file must be sent, received 0' Key: FLINK-34101 URL: https://issues.apache.org/jira/browse/FLINK-34101 Project: Flink Issue Type: Bug Components: Runtime / REST, Runtime / Web Frontend Affects Versions: 1.17.2 Reporter: blackpighe Fix For: 1.17.2 Attachments: image-2024-01-16-11-32-50-015.png, image-2024-01-16-11-35-21-582.png use flink ui upload jar, Something went wrong. !image-2024-01-16-11-32-50-015.png! message is : Server Response:org.apache.flink.runtime.rest.handler.RestHandlerException: Exactly 1 file must be sent, received 0. at org.apache.flink.runtime.webmonitor.handlers.JarUploadHandler.handleRequest(JarUploadHandler.java:80) at org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) at org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196) at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) at java.util.Optional.ifPresent(Optional.java:159) at org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) at org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) at org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:203) at org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69) at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336) at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308) at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at
[jira] [Comment Edited] (FLINK-34007) Flink Job stuck in suspend state after losing leadership in HA Mode
[ https://issues.apache.org/jira/browse/FLINK-34007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807017#comment-17807017 ] Zhenqiu Huang edited comment on FLINK-34007 at 1/16/24 3:34 AM: [~mapohl] [~wangyang0918] I am intensively testing flink 1.18. Within two days, there are users reported the job manager stuck issue in 1.17 and 1.16. 1.18 and 1.17 job instances are running in the same cluster. 1.16 is in different cluster. I attached another LeaderElector-Debug.json file that contains debug log of a flink 1.18 app. The issue happened several times: 1. due to the configmap not accessible from api sever then renew timeout exceeded 2. a failure on patch on a updated configmap The interesting part of the behavior of last several days is that job manager was not stuck but exit directly. Then, new job manager pod started correctly that is why new leader is selected in the log above. Hopefully, it is useful for your diagnosis. was (Author: zhenqiuhuang): [~mapohl] I am intensively testing flink 1.18. Within two days, there are users reported the job manager stuck issue in 1.17 and 1.16. 1.18 and 1.17 job instances are running in the same cluster. 1.16 is in different cluster. I attached another LeaderElector-Debug.json file that contains debug log of a flink 1.18 app. The issue happened several times: 1. due to the configmap not accessible from api sever then renew timeout exceeded 2. a failure on patch on a updated configmap The interesting part of the behavior of last several days is that job manager was not stuck but exit directly. Then, new job manager pod started correctly that is why new leader is selected in the log above. Hopefully, it is useful for your diagnosis. > Flink Job stuck in suspend state after losing leadership in HA Mode > --- > > Key: FLINK-34007 > URL: https://issues.apache.org/jira/browse/FLINK-34007 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.16.3, 1.17.2, 1.18.1, 1.18.2 >Reporter: Zhenqiu Huang >Priority: Major > Attachments: Debug.log, LeaderElector-Debug.json, job-manager.log > > > The observation is that Job manager goes to suspend state with a failed > container not able to register itself to resource manager after timeout. > JM Log, see attached > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34007) Flink Job stuck in suspend state after losing leadership in HA Mode
[ https://issues.apache.org/jira/browse/FLINK-34007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807017#comment-17807017 ] Zhenqiu Huang commented on FLINK-34007: --- [~mapohl] I am intensively testing flink 1.18. Within two days, there are users reported the job manager stuck issue in 1.17 and 1.16. 1.18 and 1.17 job instances are running in the same cluster. 1.16 is in different cluster. I attached another LeaderElector-Debug.json file that contains debug log of a flink 1.18 app. The issue happened several times: 1. due to the configmap not accessible from api sever then renew timeout exceeded 2. a failure on patch on a updated configmap The interesting part of the behavior of last several days is that job manager was not stuck but exit directly. Then, new job manager pod started correctly that is why new leader is selected in the log above. Hopefully, it is useful for your diagnosis. > Flink Job stuck in suspend state after losing leadership in HA Mode > --- > > Key: FLINK-34007 > URL: https://issues.apache.org/jira/browse/FLINK-34007 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.16.3, 1.17.2, 1.18.1, 1.18.2 >Reporter: Zhenqiu Huang >Priority: Major > Attachments: Debug.log, LeaderElector-Debug.json, job-manager.log > > > The observation is that Job manager goes to suspend state with a failed > container not able to register itself to resource manager after timeout. > JM Log, see attached > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-21949][table] Support ARRAY_AGG aggregate function [flink]
Jiabao-Sun commented on code in PR #23411:
URL: https://github.com/apache/flink/pull/23411#discussion_r1452891261
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/bridging/BridgingSqlAggFunction.java:
##
@@ -63,6 +64,7 @@ public final class BridgingSqlAggFunction extends
SqlAggFunction {
private final List paramTypes;
private BridgingSqlAggFunction(
+FlinkContext context,
Review Comment:
Yes, I'll revert that changes.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34007) Flink Job stuck in suspend state after losing leadership in HA Mode
[ https://issues.apache.org/jira/browse/FLINK-34007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenqiu Huang updated FLINK-34007: -- Attachment: LeaderElector-Debug.json > Flink Job stuck in suspend state after losing leadership in HA Mode > --- > > Key: FLINK-34007 > URL: https://issues.apache.org/jira/browse/FLINK-34007 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.16.3, 1.17.2, 1.18.1, 1.18.2 >Reporter: Zhenqiu Huang >Priority: Major > Attachments: Debug.log, LeaderElector-Debug.json, job-manager.log > > > The observation is that Job manager goes to suspend state with a failed > container not able to register itself to resource manager after timeout. > JM Log, see attached > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34100) Support session window table function without pulling up with window agg
[ https://issues.apache.org/jira/browse/FLINK-34100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xuyang updated FLINK-34100: --- Description: This subtask resolves the session support in ExecWindowTableFunction. And then the session window can support window agg with WindowAttachedWindowingStrategy. (was: This subtask resolves the session window agg with WindowAttachedWindowingStrategy.) > Support session window table function without pulling up with window agg > - > > Key: FLINK-34100 > URL: https://issues.apache.org/jira/browse/FLINK-34100 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: xuyang >Priority: Major > > This subtask resolves the session support in ExecWindowTableFunction. And > then the session window can support window agg with > WindowAttachedWindowingStrategy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34048) Support session window agg in table runtime instead of using legacy group window agg op
[ https://issues.apache.org/jira/browse/FLINK-34048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xuyang updated FLINK-34048: --- Description: We not only need FLINK-24024 to support session window agg in planner, but also need to support it in runtime. This subtask only resolves the session window agg with TimeAttributeWindowingStrategy. was:We not only need FLINK-24024 to support session window agg in planner, but also need to support it in runtime. > Support session window agg in table runtime instead of using legacy group > window agg op > --- > > Key: FLINK-34048 > URL: https://issues.apache.org/jira/browse/FLINK-34048 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Affects Versions: 1.19.0 >Reporter: xuyang >Priority: Major > > We not only need FLINK-24024 to support session window agg in planner, but > also need to support it in runtime. > This subtask only resolves the session window agg with > TimeAttributeWindowingStrategy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34100) Support session window table function without pulling up with window agg
[ https://issues.apache.org/jira/browse/FLINK-34100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xuyang updated FLINK-34100: --- Description: This subtask resolves the session window agg with WindowAttachedWindowingStrategy. > Support session window table function without pulling up with window agg > - > > Key: FLINK-34100 > URL: https://issues.apache.org/jira/browse/FLINK-34100 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: xuyang >Priority: Major > > This subtask resolves the session window agg with > WindowAttachedWindowingStrategy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34070][test] Adds dedicated tests in MiniClusterITCase for scenarios where there are not enough slots available. [flink]
jiangxin369 commented on code in PR #24082:
URL: https://github.com/apache/flink/pull/24082#discussion_r1452878415
##
tools/azure-pipelines/build-apache-repo.yml:
##
@@ -73,6 +73,17 @@ stages:
run_end_to_end: false
container: flink-build-container
jdk: 8
+ - template: jobs-template.yml
+parameters:
+ stage_name: ci-adaptive-scheduler
+ test_pool_definition:
+name: Default
+ e2e_pool_definition:
+vmImage: 'ubuntu-20.04'
+ environment: PROFILE="-Penable-adaptive-scheduler"
Review Comment:
Shall we add the `-Dflink.hadoop.version=2.10.2` in the profile to keep
consistency with other stages? We can also remove the profile from other stages
If `2.10.2` is the default value.
##
flink-runtime/src/test/java/org/apache/flink/runtime/minicluster/MiniClusterITCase.java:
##
@@ -149,7 +174,15 @@ private void runHandleJobsWhenNotEnoughSlots(final
JobGraph jobGraph) throws Exc
try (final MiniCluster miniCluster = new MiniCluster(cfg)) {
miniCluster.start();
-miniCluster.executeJobBlocking(jobGraph);
+assertThatThrownBy(() -> miniCluster.executeJobBlocking(jobGraph))
+.isInstanceOf(JobExecutionException.class)
+.hasMessageContaining("Job execution failed")
+.extracting(Throwable::getCause)
+.extracting(FlinkAssertions::chainOfCauses,
FlinkAssertions.STREAM_THROWABLE)
+.anySatisfy(
+cause ->
+assertThat(cause)
+
.isInstanceOf(NoResourceAvailableException.class));
Review Comment:
Would it be more explicit if we assert the TimeoutException is thrown when
running `testHandlingNotEnoughSlotsThroughTimeout` with the default scheduler?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-34100) Support session window table function without pulling up with window agg
xuyang created FLINK-34100: -- Summary: Support session window table function without pulling up with window agg Key: FLINK-34100 URL: https://issues.apache.org/jira/browse/FLINK-34100 Project: Flink Issue Type: Sub-task Components: Table SQL / Runtime Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33988][configuration] Fix the invalid configuration when using initialized root logger level on yarn deployment mode [flink]
X-czh commented on PR #24042: URL: https://github.com/apache/flink/pull/24042#issuecomment-1892997269 Hi @1996fanrui, I've verified that the current release of Flink supports modifying the logger level in both Native K8s and Standalone mode. Since YARN prepares the container launching context in a special way, we'll need to do some special adaptation anyway. LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33988) Invalid configuration when using initialized root logger level on yarn application mode
[
https://issues.apache.org/jira/browse/FLINK-33988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807014#comment-17807014
]
Zhanghao Chen commented on FLINK-33988:
---
[~RocMarshal] Thanks for reporting this. You are right, the impl of FLINK-33166
won't work under YARN Application mode. The impl of FLINK-33166 replies on the
preparing the starting context in pure shell code (Standalone & Native K8s
follows this track). However, YARN application mode prepares the starting
context in Java code
(flink-yarn/YarnClusterDescriptor#setupApplicationMasterContainer &
flink-yarn/Utils#createTaskExecutorContext, the logging-related config is
assembled in YarnLogConfigUtil). It seems thatthe option {{env.log.max}} does
not take effect under YARN as well.
> Invalid configuration when using initialized root logger level on yarn
> application mode
> ---
>
> Key: FLINK-33988
> URL: https://issues.apache.org/jira/browse/FLINK-33988
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration
>Affects Versions: 1.19.0
>Reporter: RocMarshal
>Assignee: RocMarshal
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> relevant https://issues.apache.org/jira/browse/FLINK-33166
> When I set env. log. level=DEBUG and start the flink job by yarn application
> mode, the logs of TM and JM are still INFO.
> Preliminary inference is that it is ROOT_ LOG_ LEVEL value transmission link
> is not complete enough.
> So I used the following configuration:
> containerized. taskmanager. env. ROOT_ LOG_ LEVEL=DEBUG
> containerized. master. env. ROOT_ LOG_ LEVEL=DEBUG
>
> When starting the job by yarn application mode, TM and JM can output DEBUG
> level logs.
>
> Repair ideas:
> Fill the value of *env. log. level* into the Flink configuration by
> *containerized. xxx. env. ROOT_ LOG_ LEVEL* before obtaining the environment
> variable for the container
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33929][jdbc-connector] Support JDBC String field read Fragment read [flink-connector-jdbc]
zhilinli123 commented on PR #87: URL: https://github.com/apache/flink-connector-jdbc/pull/87#issuecomment-1892982217 > > @zhilinli123 I still don't understand the Jira ticket, so I can't really review it. Perhaps @snuyanzin or @eskabetxe understand the goal of this PR. > > For what I can assume, @zhilinli123 are trying to use scan.partition on a table that is hash partitioned by a string field.. > > My understanding of scan.partition is that only divides the query on multiples query based on params of scan.partition but is not using partitioning of table, it only adds "field BETWEEN ? AND ?" in query, and for that it only allow numeric, date, or timestamp column.. You're right, but slicing string hash md5 doesn't need to be done BETWEEN? AND ? It's much more efficient to check this shard directly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-29802) ChangelogStateBackend supports native savepoint
[ https://issues.apache.org/jira/browse/FLINK-29802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hangxiang Yu reassigned FLINK-29802: Assignee: Hangxiang Yu > ChangelogStateBackend supports native savepoint > --- > > Key: FLINK-29802 > URL: https://issues.apache.org/jira/browse/FLINK-29802 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Hangxiang Yu >Assignee: Hangxiang Yu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-34078) Move InternalKeyContext classes from o.a.f.runtime.state.heap to o.a.f.runtime.state package
[ https://issues.apache.org/jira/browse/FLINK-34078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hangxiang Yu resolved FLINK-34078. -- Fix Version/s: 1.19.0 Resolution: Fixed merged 6bdb4f75 into master > Move InternalKeyContext classes from o.a.f.runtime.state.heap to > o.a.f.runtime.state package > > > Key: FLINK-34078 > URL: https://issues.apache.org/jira/browse/FLINK-34078 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Jinzhong Li >Assignee: Jinzhong Li >Priority: Minor > Labels: pull-request-available > Fix For: 1.19.0 > > Attachments: image-2024-01-15-12-57-12-667.png > > > h3. Motication: > When Rocksdb statebackend throws a keyGroup check illegal exception, > the exception stack contains the heap stateBackend scoped class, which looks > so strange to user. > !image-2024-01-15-12-57-12-667.png|width=555,height=68! > h3. Proposed changes: > InternalKeyContext and InternalKeyContextImpl are commonly used by all state > backends (heap/rocksdb/changelog), they should be moved from > org.apache.flink.runtime.state.heap package to org.apache.flink.runtime.state > package. > h3. Compatibility: > InternalKeyContext is annotated with @Internal, so this change has no > compatibility issues. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33689][table-runtime] Fix JsonObjectAggFunction can't retract records when enabling LocalGlobal. [flink]
xishuaidelin commented on code in PR #23827:
URL: https://github.com/apache/flink/pull/23827#discussion_r1452864030
##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/AggregateITCase.scala:
##
@@ -1916,4 +1916,44 @@ class AggregateITCase(aggMode: AggMode, miniBatch:
MiniBatchMode, backend: State
)
assertThat(sink.getRetractResults.sorted).isEqualTo(expected.sorted)
}
+
+ @TestTemplate
+ def testGroupJsonObjectAggWithRetract(): Unit = {
+val data = new mutable.MutableList[(Long, String, Long)]
+data.+=((2L, "Hallo", 2L))
Review Comment:
Hi lincoln, Thanks for your response. I have added the tests with nested agg
and simplify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. [flink-connector-hbase]
MOBIN-F commented on code in PR #36:
URL:
https://github.com/apache/flink-connector-hbase/pull/36#discussion_r1452863933
##
flink-connector-hbase-base/src/main/java/org/apache/flink/connector/hbase/sink/RowDataToMutationConverter.java:
##
@@ -43,23 +43,26 @@ public class RowDataToMutationConverter implements
HBaseMutationConverter DYNAMIC_TABLE =
+ConfigOptions.key("dynamic.table")
Review Comment:
yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33689][table-runtime] Fix JsonObjectAggFunction can't retract records when enabling LocalGlobal. [flink]
xishuaidelin commented on code in PR #23827:
URL: https://github.com/apache/flink/pull/23827#discussion_r1452862528
##
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/aggregate/JsonObjectAggFunction.java:
##
@@ -128,6 +137,15 @@ public void merge(Accumulator acc, Iterable
others) throws Exceptio
assertKeyNotPresent(acc, key);
acc.map.put(key, other.map.get(key));
}
+for (final StringData key : other.retractMap.keys()) {
Review Comment:
Hi xuyang, Thanks for your comments. if +U is eliminated by -U, It means two
messages has same key and value. It is an expected behaviour. However, you
reminded me that the implementation here needs to be changed. Retraction is
supposed to be done first in merge function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. [flink-connector-hbase]
MOBIN-F commented on code in PR #36: URL: https://github.com/apache/flink-connector-hbase/pull/36#discussion_r1452862759 ## flink-connector-hbase-base/src/main/java/org/apache/flink/connector/hbase/util/HBaseSerde.java: ## Review Comment: I agree with this point of view. If you have no objection to the dynamic-table parameter, I am willing to do this part of the refactoring work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34099) CheckpointIntervalDuringBacklogITCase.testNoCheckpointDuringBacklog is unstable on AZP
Yu Chen created FLINK-34099:
---
Summary:
CheckpointIntervalDuringBacklogITCase.testNoCheckpointDuringBacklog is unstable
on AZP
Key: FLINK-34099
URL: https://issues.apache.org/jira/browse/FLINK-34099
Project: Flink
Issue Type: Bug
Affects Versions: 1.19.0
Reporter: Yu Chen
This build [Pipelines - Run 20240115.30 logs
(azure.com)|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56403=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba]
fails as
{code:java}
Jan 15 18:29:51 18:29:51.938 [ERROR]
org.apache.flink.test.checkpointing.CheckpointIntervalDuringBacklogITCase.testNoCheckpointDuringBacklog
-- Time elapsed: 2.022 s <<< FAILURE!
Jan 15 18:29:51 org.opentest4j.AssertionFailedError:
Jan 15 18:29:51
Jan 15 18:29:51 expected: 0
Jan 15 18:29:51 but was: 1
Jan 15 18:29:51 at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jan 15 18:29:51 at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Jan 15 18:29:51 at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jan 15 18:29:51 at
org.apache.flink.test.checkpointing.CheckpointIntervalDuringBacklogITCase.testNoCheckpointDuringBacklog(CheckpointIntervalDuringBacklogITCase.java:141)
Jan 15 18:29:51 at java.lang.reflect.Method.invoke(Method.java:498)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33971]Specifies whether to use HBase table that supports dynamic columns. [flink-connector-hbase]
MOBIN-F commented on PR #36: URL: https://github.com/apache/flink-connector-hbase/pull/36#issuecomment-1892973838 > > the main question is that is it a good idea to introduce platform-specific features and flags to the IMO common upstream connector.主要问题是，将特定于平台的功能和标志引入 IMO 通用上游连接器是否是一个好主意。 > > If this isn't something that isn't supported in open source HBase itself, then at least we should first have a discussion on this topic in the Dev mailing list.如果开源 HBase 本身不支持这一点，那么至少我们应该首先在 Dev 邮件列表中讨论这个话题。 @MartijnVisser @ferenc-csaky open source hbase is supported. We have used this function internally online. The version is also open source hbase and has nothing to do with the cloud platform. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-29802][state] Changelog supports native savepoint [flink]
masteryhx commented on PR #22744: URL: https://github.com/apache/flink/pull/22744#issuecomment-1892973498 @rkhachatryan Could you help to take a review again ? cc @Zakelly You may also be interested in this. Since Changelog doesn't support NO_CLAIM mode yet, we could let the native savepoint usable so that users could have a quick backup of snapshot to other jobs (see [comment](https://issues.apache.org/jira/browse/FLINK-25322?focusedCommentId=17738073=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17738073)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33668) Decoupling Shuffle network memory and job topology
[ https://issues.apache.org/jira/browse/FLINK-33668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xin updated FLINK-33668: -- Description: With FLINK-30469 and FLINK-31643, we have decoupled the shuffle network memory and the parallelism of tasks by limiting the number of buffers for each InputGate and ResultPartition in Hybrid Shuffle. However, when a task contains multiple ResultPartitions or many shuffle tasks are running simultaneously on the same TaskManager, "Insufficient number of network buffers" errors would still occur. This usually happens when Slot Sharing Group is enabled or a TaskManager contains multiple slots. We want to make sure that the TaskManager does not encounter "Insufficient number of network buffers" even if there are dozens of InputGates and ResultPartitions running on the same TaskManager simultaneously. I have given this some thought, and here is my rough proposal. 1. InputGate or ResultPartition only apply for buffers from LocalBufferPool, which means that InputGate will no longer ask for exclusive buffers from NetworkBufferPool directly. 2. When creating a LocalBufferPool, we need to specify the maximum, minimum, and expected number of buffers. Whenever a new LBP is created or destroyed, a redistribution will occur, to adjust the buffer count of all LocalBufferPools using the expected value as a weight and between the minimum and maximum values. According to the test, the minimum value can be set to 4 to make the Flink Job work despite the possibility of lower performance. With this minimum value, a task with 20 shuffle edges needs only 5MB of memory to avoid "insufficient network buffer" error. 3. During runtime, InputGate and ResultPartition both calculate the number of buffers used by their internal data structures based on the pool size of their corresponding LocalBufferPool, such as the exclusive buffer queue of InputGate and BufferAccumulator of ResultPartition. was: With FLINK-30469 and FLINK-31643, we have decoupled the shuffle network memory and the parallelism of tasks by limiting the number of buffers for each InputGate and ResultPartition in Hybrid Shuffle. However, when too many shuffle tasks are running simultaneously on the same TaskManager, "Insufficient number of network buffers" errors would still occur. This usually happens when Slot Sharing Group is enabled or a TaskManager contains multiple slots. We want to make sure that the TaskManager does not encounter "Insufficient number of network buffers" even if there are dozens of InputGates and ResultPartitions running on the same TaskManager simultaneously. I have given this some thought, and here is my rough proposal. 1. InputGate or ResultPartition only apply for buffers from LocalBufferPool, which means that InputGate will no longer ask for exclusive buffers from NetworkBufferPool directly. 2. When creating a LocalBufferPool, we need to specify the maximum, minimum, and expected number of buffers. Whenever a new LBP is created or destroyed, a redistribution will occur, to adjust the buffer count of all LocalBufferPools using the expected value as a weight and between the minimum and maximum values. According to the test, the minimum value can be set to 4 to make the Flink Job work despite the possibility of lower performance. With this minimum value, a task with 20 shuffle edges needs only 5MB of memory to avoid "insufficient network buffer" error. 3. During runtime, InputGate and ResultPartition both calculate the number of buffers used by their internal data structures based on the pool size of their corresponding LocalBufferPool, such as the exclusive buffer queue of InputGate and BufferAccumulator of ResultPartition. > Decoupling Shuffle network memory and job topology > -- > > Key: FLINK-33668 > URL: https://issues.apache.org/jira/browse/FLINK-33668 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Reporter: Jiang Xin >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > With FLINK-30469 and FLINK-31643, we have decoupled the shuffle network > memory and the parallelism of tasks by limiting the number of buffers for > each InputGate and ResultPartition in Hybrid Shuffle. However, when a task > contains multiple ResultPartitions or many shuffle tasks are running > simultaneously on the same TaskManager, "Insufficient number of network > buffers" errors would still occur. This usually happens when Slot Sharing > Group is enabled or a TaskManager contains multiple slots. > We want to make sure that the TaskManager does not encounter "Insufficient > number of network buffers" even if there are dozens of InputGates and > ResultPartitions running on the same TaskManager
Re: [PR] [FLINK-34078][state]Move InternalKeyContext classes from o.a.f.runtim… [flink]
ljz2051 commented on PR #24093: URL: https://github.com/apache/flink/pull/24093#issuecomment-1892971770 @masteryhx Would you please help to review this pr? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34007) Flink Job stuck in suspend state after losing leadership in HA Mode
[
https://issues.apache.org/jira/browse/FLINK-34007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807009#comment-17807009
]
Yang Wang commented on FLINK-34007:
---
{quote}At least based on the reports of this Jira issue, there must have been
an incident (which caused the lease to not be renewed)
{quote}
I am afraid we could not get this conclusion before we have the K8s APIServer
audit logs to verify that the lease annotation did not get renewed. Because it
could also happen that the lease annotation get renewed normally while the
onStartLeading callback is not executed somehow.
{quote}Therefore, the issue should exist in the entire version range [5.12.3,
6.6.2].
{quote}
If this issue only happened in the Flink 1.18, then it should be related with
the fabric8 K8s client 6.6.2 behavior change. Otherwise, we still have not find
the root cause.
You are right. The slight difference in the revocation protocol in the
[FLIP-285|https://cwiki.apache.org/confluence/display/FLINK/FLIP-285%3A+Refactoring+LeaderElection+to+make+Flink+support+multi-component+leader+election+out-of-the-box]
changes about clear the leader information in ConfigMap is not related with
this issue.
BTW, if we know how to reproduce this issue, it will be easier to find the root
cause. Because we might also need the K8s APIServer audit log to do some deep
analysis.
> Flink Job stuck in suspend state after losing leadership in HA Mode
> ---
>
> Key: FLINK-34007
> URL: https://issues.apache.org/jira/browse/FLINK-34007
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.3, 1.17.2, 1.18.1, 1.18.2
>Reporter: Zhenqiu Huang
>Priority: Major
> Attachments: Debug.log, job-manager.log
>
>
> The observation is that Job manager goes to suspend state with a failed
> container not able to register itself to resource manager after timeout.
> JM Log, see attached
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34089] Verify that the subscribed topics match the assigned topics [flink-connector-kafka]
Tan-JiaLiang commented on PR #77: URL: https://github.com/apache/flink-connector-kafka/pull/77#issuecomment-1892967906 @tzulitai Hi gordon, can you take a look when you are free? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33950) Update max aggregate functions to new type system
[ https://issues.apache.org/jira/browse/FLINK-33950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807008#comment-17807008 ] Jacky Lau commented on FLINK-33950: --- hi [~martijnvisser] [~Zakelly] [~dwysakowicz] , i found it will have 69 test failed https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/55946/logs/104. and if we change the plan to make the test passed . But some test we should not change, because some unit test is for construct a hash agg to test some other rules, if we change it to sort agg, the test will not take effective. > Update max aggregate functions to new type system > - > > Key: FLINK-33950 > URL: https://issues.apache.org/jira/browse/FLINK-33950 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Jacky Lau >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33386][runtime] Support tasks balancing at slot level for Default Scheduler [flink]
KarmaGYZ commented on PR #23635: URL: https://github.com/apache/flink/pull/23635#issuecomment-1892964860 > @RocMarshal Just be curious about the progress, does this PR still wait for some comments to be addressed before it could be merged? This PR is in progress now. We plan to merge it after the complete Task Balancing feature is implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33819] support set CompressionType for RocksDBStateBackend [flink]
masteryhx commented on code in PR #24072:
URL: https://github.com/apache/flink/pull/24072#discussion_r1452854496
##
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBConfigurableOptions.java:
##
@@ -139,6 +149,25 @@ public class RocksDBConfigurableOptions implements
Serializable {
NONE.name(),
LEVEL.name()));
+public static final ConfigOption COMPRESSION_TYPE =
Review Comment:
+1 to the idea of @pnowojski
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34098) Not enough Azure Pipeline CI runners available?
[ https://issues.apache.org/jira/browse/FLINK-34098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807004#comment-17807004 ] lincoln lee commented on FLINK-34098: - [~jingge] Can you help check it? > Not enough Azure Pipeline CI runners available? > --- > > Key: FLINK-34098 > URL: https://issues.apache.org/jira/browse/FLINK-34098 > Project: Flink > Issue Type: Bug > Components: Build System / CI >Affects Versions: 1.18.0, 1.17.2, 1.19.0 >Reporter: Matthias Pohl >Priority: Critical > > CI takes longer than usually. There might be an issue with the amount of > runners (on the Alibaba VMs)? -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32659] Fix the connection leak due to exception [flink-connector-jdbc]
Aitozi closed pull request #71: [FLINK-32659] Fix the connection leak due to exception URL: https://github.com/apache/flink-connector-jdbc/pull/71 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34088) fix the problems with special table name characters of postgres and oracle and sqlserver.
[ https://issues.apache.org/jira/browse/FLINK-34088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807002#comment-17807002 ] blackpighe commented on FLINK-34088: This problem has persisted in other releases as well, but take the new project's 3.1.1 as an example > fix the problems with special table name characters of postgres and oracle > and sqlserver. > -- > > Key: FLINK-34088 > URL: https://issues.apache.org/jira/browse/FLINK-34088 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: jdbc-3.1.1 >Reporter: blackpighe >Priority: Major > Fix For: jdbc-3.1.1 > > Attachments: image-2024-01-15-17-10-57-586.png, > image-2024-01-15-17-11-11-136.png, image-2024-01-15-17-13-16-448.png > > Original Estimate: 48h > Remaining Estimate: 48h > > > !image-2024-01-15-17-11-11-136.png! > [https://github.com/apache/flink-connector-jdbc/] > The new version of flink jdbc is stand-alone and does not have an open issue > entry. > > The FLink-JDBC-connector does not handle special table names for oracle and > postgres and sqlserver, when the table name is a special character such as > default. jdbc will report an error. hopefully giving me a chance to fix the > problem, I can provide a commit > > > !image-2024-01-15-17-13-16-448.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34088) fix the problems with special table name characters of postgres and oracle and sqlserver.
[ https://issues.apache.org/jira/browse/FLINK-34088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807001#comment-17807001 ] blackpighe commented on FLINK-34088: i edit and update the issue, by use english > fix the problems with special table name characters of postgres and oracle > and sqlserver. > -- > > Key: FLINK-34088 > URL: https://issues.apache.org/jira/browse/FLINK-34088 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: jdbc-3.1.1 >Reporter: blackpighe >Priority: Major > Fix For: jdbc-3.1.1 > > Attachments: image-2024-01-15-17-10-57-586.png, > image-2024-01-15-17-11-11-136.png, image-2024-01-15-17-13-16-448.png > > Original Estimate: 48h > Remaining Estimate: 48h > > > !image-2024-01-15-17-11-11-136.png! > [https://github.com/apache/flink-connector-jdbc/] > The new version of flink jdbc is stand-alone and does not have an open issue > entry. > > The FLink-JDBC-connector does not handle special table names for oracle and > postgres and sqlserver, when the table name is a special character such as > default. jdbc will report an error. hopefully giving me a chance to fix the > problem, I can provide a commit > > > !image-2024-01-15-17-13-16-448.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34088) fix the problems with special table name characters of postgres and oracle and sqlserver.
[ https://issues.apache.org/jira/browse/FLINK-34088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] blackpighe updated FLINK-34088: --- Fix Version/s: jdbc-3.1.1 Affects Version/s: jdbc-3.1.1 (was: 1.17.2) Description: !image-2024-01-15-17-11-11-136.png! [https://github.com/apache/flink-connector-jdbc/] The new version of flink jdbc is stand-alone and does not have an open issue entry. The FLink-JDBC-connector does not handle special table names for oracle and postgres and sqlserver, when the table name is a special character such as default. jdbc will report an error. hopefully giving me a chance to fix the problem, I can provide a commit !image-2024-01-15-17-13-16-448.png! was: !image-2024-01-15-17-11-11-136.png! jdbc使用特殊表名需要进行转义，发现1.17版本，jdbc connector单独移出去成为了一个新项目，我希望给一个commit的机会，来修复这个问题，现在https://github.com/apache/flink-connector-jdbc/这个项目没有create issue的入口 !image-2024-01-15-17-13-16-448.png! > fix the problems with special table name characters of postgres and oracle > and sqlserver. > -- > > Key: FLINK-34088 > URL: https://issues.apache.org/jira/browse/FLINK-34088 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC >Affects Versions: jdbc-3.1.1 >Reporter: blackpighe >Priority: Major > Fix For: jdbc-3.1.1 > > Attachments: image-2024-01-15-17-10-57-586.png, > image-2024-01-15-17-11-11-136.png, image-2024-01-15-17-13-16-448.png > > Original Estimate: 48h > Remaining Estimate: 48h > > > !image-2024-01-15-17-11-11-136.png! > [https://github.com/apache/flink-connector-jdbc/] > The new version of flink jdbc is stand-alone and does not have an open issue > entry. > > The FLink-JDBC-connector does not handle special table names for oracle and > postgres and sqlserver, when the table name is a special character such as > default. jdbc will report an error. hopefully giving me a chance to fix the > problem, I can provide a commit > > > !image-2024-01-15-17-13-16-448.png! > -- This message was sent by Atlassian Jira (v8.20.10#820010)
