[GitHub] [flink] Aitozi commented on pull request #21174: [FLINK-29557] Fix the SinkOperator with OutputFormatFunction is ignor…
Aitozi commented on PR #21174: URL: https://github.com/apache/flink/pull/21174#issuecomment-1294497007 Maybe it's caused by this: https://issues.apache.org/jira/browse/FLINK-29787 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Aitozi commented on pull request #21174: [FLINK-29557] Fix the SinkOperator with OutputFormatFunction is ignor…
Aitozi commented on PR #21174: URL: https://github.com/apache/flink/pull/21174#issuecomment-1294495106 Hi @gaoyunhaii , can you help me understand the CI failure? > [ERROR] Failed to execute goal io.github.zentol.japicmp:japicmp-maven-plugin:0.16.0_m325:cmp (default) on project flink-core: There is at least one incompatibility: org.apache.flink.api.connector.source.SourceReader.pauseOrResumeSplits(java.util.Collection,java.util.Collection):METHOD_NEW_DEFAULT -> [Help 1] The reported method seems not related to this PR, a little confused for the result -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26390) InputStream is not closed at CliClient#executeFile
[
https://issues.apache.org/jira/browse/FLINK-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-26390:
--
Fix Version/s: 1.17.0
> InputStream is not closed at CliClient#executeFile
> --
>
> Key: FLINK-26390
> URL: https://issues.apache.org/jira/browse/FLINK-26390
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / Client
>Affects Versions: 1.15.0
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.17.0
>
>
> There is {{InputStream}} created however not closed
> https://github.com/apache/flink/blob/e2463ddace26601c6442dbaa68c198a27c9cfbcf/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java#L349-L364
> Another thing, which is easy to fix: it seems there is no need to create
> completer object for {{CliClient#executeFile}} since it makes sense only in
> interactive mode
> It seems the reason is https://github.com/apache/flink/pull/18363/files
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29748) Expose the optimize phase in the connector context
[
https://issues.apache.org/jira/browse/FLINK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625440#comment-17625440
]
Aitozi commented on FLINK-29748:
As I know, the \{{getScanRuntimeProvider}} will be invoked serval times during
the optimize, eg:
*
org.apache.flink.table.planner.connectors.DynamicSourceUtils#validateScanSource
*
org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecTableSourceScan#translateToPlanInternal
So, to the connector, it do not know whether the optimization is finished.
> Expose the optimize phase in the connector context
> --
>
> Key: FLINK-29748
> URL: https://issues.apache.org/jira/browse/FLINK-29748
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner, Table SQL / Runtime
>Reporter: Aitozi
>Priority: Minor
>
> Currently, in the connector it can not know whether the whole optimize is
> finished.
> When the optimize finished, the all information is static, eg: the reading
> partitions. If I want to validate the final optimized result (like whether
> the reading partition is too much or empty), it needs the context of what is
> the current phase. I think the {{ScanContext}} is ok to expose this
> information.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-26390) InputStream is not closed at CliClient#executeFile
[
https://issues.apache.org/jira/browse/FLINK-26390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl resolved FLINK-26390.
---
Resolution: Fixed
master: 74c0ace01b61f18cda5c13abf0b940fe81f13e8a
> InputStream is not closed at CliClient#executeFile
> --
>
> Key: FLINK-26390
> URL: https://issues.apache.org/jira/browse/FLINK-26390
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / Client
>Affects Versions: 1.15.0
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
>
> There is {{InputStream}} created however not closed
> https://github.com/apache/flink/blob/e2463ddace26601c6442dbaa68c198a27c9cfbcf/flink-table/flink-sql-client/src/main/java/org/apache/flink/table/client/cli/CliClient.java#L349-L364
> Another thing, which is easy to fix: it seems there is no need to create
> completer object for {{CliClient#executeFile}} since it makes sense only in
> interactive mode
> It seems the reason is https://github.com/apache/flink/pull/18363/files
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] XComp merged pull request #18927: [FLINK-26390] Use try resourses for CliClient#executeFile
XComp merged PR #18927: URL: https://github.com/apache/flink/pull/18927 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #167: [FLINK-29786] VarianceThresholdSelector Uses FeaturesCol Incorrectly
lindong28 commented on code in PR #167:
URL: https://github.com/apache/flink-ml/pull/167#discussion_r1007646156
##
flink-ml-lib/src/main/java/org/apache/flink/ml/common/param/HasFeaturesCol.java:
##
@@ -24,6 +24,14 @@
import org.apache.flink.ml.param.WithParams;
/** Interface for the shared featuresCol param. */
+
+/**
+ * Interface for the shared featuresCol param.
Review Comment:
Should we remove the redundant description above?
And it might be simpler and softer to use the following doc:
```
/**
* Interface for the shared featuresCol param.
*
* {@link HasFeaturesCol} is typically used for {@link Stage}s that
implement {@link HasLabelCol}. It is preferred
* to use {@link HasInputCol} for other cases.
*/
```
Please also update the corresponding python doc if we make the change here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #166: [FLINK-29598] Add Estimator and Transformer for Imputer
yunfengzhou-hub commented on code in PR #166:
URL: https://github.com/apache/flink-ml/pull/166#discussion_r1007561913
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/imputer/ImputerParams.java:
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.imputer;
+
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.StringParam;
+
+/**
+ * Params of {@link Imputer}.
+ *
+ * @param The class type of this instance.
+ */
+public interface ImputerParams extends ImputerModelParams {
Review Comment:
Spark's `Imputer` also has a `relativeError` parameter. Should we also add
this parameter to Flink ML?
##
flink-ml-core/src/main/java/org/apache/flink/ml/param/DoubleParam.java:
##
@@ -32,4 +34,12 @@ public DoubleParam(
public DoubleParam(String name, String description, Double defaultValue) {
this(name, description, defaultValue, ParamValidators.alwaysTrue());
}
+
+@Override
+public Double jsonDecode(Object json) throws IOException {
+if (json instanceof String && json.equals(String.valueOf(Double.NaN)))
{
+return Double.NaN;
+}
+return (Double) json;
Review Comment:
How about the following implementation?
```java
if (json instanceof String) {
return Double.valueOf((String) json);
}
return (Double) json;
```
This applies to `Double.NaN` as well as other special values.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/imputer/Imputer.java:
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.imputer;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.ml.api.Estimator;
+import org.apache.flink.ml.common.datastream.DataStreamUtils;
+import org.apache.flink.ml.common.util.QuantileSummary;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * The Imputer estimator completes missing values in a dataset. Missing values
can be imputed using
Review Comment:
nit: it might be better to use "bounded stream" instead of "dataset", as
dataset has a specific meaning in flink.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/imputer/ImputerModel.java:
##
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ *
[jira] [Comment Edited] (FLINK-29541) [JUnit5 Migration] Module: flink-table-planner
[ https://issues.apache.org/jira/browse/FLINK-29541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625437#comment-17625437 ] Lijie Wang edited comment on FLINK-29541 at 10/28/22 5:37 AM: -- Sure, just assigned to you :). It would be better to create a separate issue to track these abstract base classes, just like what you did in FLINK-28542 was (Author: wanglijie95): Sure, just assigned to you :). It would be better to create a separate issue for these abstract base classes to track them, just like what you did in FLINK-28542 > [JUnit5 Migration] Module: flink-table-planner > -- > > Key: FLINK-29541 > URL: https://issues.apache.org/jira/browse/FLINK-29541 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner, Tests >Reporter: Lijie Wang >Assignee: Ryan Skraba >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29541) [JUnit5 Migration] Module: flink-table-planner
[ https://issues.apache.org/jira/browse/FLINK-29541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijie Wang reassigned FLINK-29541: -- Assignee: Ryan Skraba > [JUnit5 Migration] Module: flink-table-planner > -- > > Key: FLINK-29541 > URL: https://issues.apache.org/jira/browse/FLINK-29541 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner, Tests >Reporter: Lijie Wang >Assignee: Ryan Skraba >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29541) [JUnit5 Migration] Module: flink-table-planner
[ https://issues.apache.org/jira/browse/FLINK-29541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625437#comment-17625437 ] Lijie Wang commented on FLINK-29541: Sure, just assigned to you :). It would be better to create a separate issue for these abstract base classes to track them, just like what you did in FLINK-28542 > [JUnit5 Migration] Module: flink-table-planner > -- > > Key: FLINK-29541 > URL: https://issues.apache.org/jira/browse/FLINK-29541 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner, Tests >Reporter: Lijie Wang >Assignee: Ryan Skraba >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29565) In Flink per job mode, the logs printed by taskManager on the web UI will not be highlighted, because the log contents are annotated due to special symbols, which will
[ https://issues.apache.org/jira/browse/FLINK-29565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao reassigned FLINK-29565: --- Assignee: wangshiwei > In Flink per job mode, the logs printed by taskManager on the web UI will not > be highlighted, because the log contents are annotated due to special > symbols, which will affect the use experience. For more information, see Fig > > > Key: FLINK-29565 > URL: https://issues.apache.org/jira/browse/FLINK-29565 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.14.3 >Reporter: wangshiwei >Assignee: wangshiwei >Priority: Minor > Attachments: image-2022-10-10-18-40-27-721.png, > image-2022-10-10-18-43-31-897.png, image-2022-10-10-18-43-53-713.png, > image-2022-10-10-18-45-17-228.png, image-2022-10-10-19-02-29-796.png, > image-2022-10-10-19-03-27-670.png > > > > !image-2022-10-10-19-03-27-670.png|width=580,height=317! > !image-2022-10-10-18-43-53-713.png|width=726,height=47! > This kind of '/*' content will appear in the print log of the > logEnvironmentInfo method in the EnvironmentInformation class. The following > logs will be commented out without highlighting > *verification* > !image-2022-10-10-18-45-17-228.png|width=880,height=161! > After manually printing '*/' in the business code, the log is normal > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29565) In Flink per job mode, the logs printed by taskManager on the web UI will not be highlighted, because the log contents are annotated due to special symbols, which will
[ https://issues.apache.org/jira/browse/FLINK-29565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625435#comment-17625435 ] Yun Gao commented on FLINK-29565: - Thanks [~1336399775] for helping fix the issue! I have assigned the issue to you. > In Flink per job mode, the logs printed by taskManager on the web UI will not > be highlighted, because the log contents are annotated due to special > symbols, which will affect the use experience. For more information, see Fig > > > Key: FLINK-29565 > URL: https://issues.apache.org/jira/browse/FLINK-29565 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.14.3 >Reporter: wangshiwei >Priority: Minor > Attachments: image-2022-10-10-18-40-27-721.png, > image-2022-10-10-18-43-31-897.png, image-2022-10-10-18-43-53-713.png, > image-2022-10-10-18-45-17-228.png, image-2022-10-10-19-02-29-796.png, > image-2022-10-10-19-03-27-670.png > > > > !image-2022-10-10-19-03-27-670.png|width=580,height=317! > !image-2022-10-10-18-43-53-713.png|width=726,height=47! > This kind of '/*' content will appear in the print log of the > logEnvironmentInfo method in the EnvironmentInformation class. The following > logs will be commented out without highlighting > *verification* > !image-2022-10-10-18-45-17-228.png|width=880,height=161! > After manually printing '*/' in the business code, the log is normal > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #162: [FLINK-29593] Add QuantileSummary to help calculate approximate quantiles
lindong28 commented on code in PR #162:
URL: https://github.com/apache/flink-ml/pull/162#discussion_r1007633262
##
flink-ml-lib/src/main/java/org/apache/flink/ml/common/util/QuantileSummary.java:
##
@@ -0,0 +1,400 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Helper class to compute an approximate quantile summary. This
implementation is based on the
+ * algorithm proposed in the paper: "Space-efficient Online Computation of
Quantile Summaries" by
+ * Greenwald, Michael and Khanna, Sanjeev.
(https://doi.org/10.1145/375663.375670)
+ */
+@Internal
+public class QuantileSummary implements Serializable {
+
+/** The target relative error. */
+private final double relativeError;
+
+/**
+ * The compression threshold. After the internal buffer of statistics
crosses this size, it
+ * attempts to compress the statistics together.
+ */
+private final int compressThreshold;
+
+/** The count of all the elements inserted to be calculated. */
+private final long count;
+
+/** A buffer of quantile statistics. */
+private final List sampled;
+
+/** The default size of head buffer. */
+private static final int DEFAULT_HEAD_SIZE = 5;
Review Comment:
`private static final` variables are typically put above `private final`
variables.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/common/util/QuantileSummary.java:
##
@@ -0,0 +1,400 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Helper class to compute an approximate quantile summary. This
implementation is based on the
+ * algorithm proposed in the paper: "Space-efficient Online Computation of
Quantile Summaries" by
+ * Greenwald, Michael and Khanna, Sanjeev.
(https://doi.org/10.1145/375663.375670)
+ */
+@Internal
Review Comment:
According to the Java doc of `@Internal`, it is an "Annotation to mark
methods within stable, public APIs as an internal developer API".
Thus it seems unnecessary to add this annotation.
@zhipeng93 Could you confirm that we should remove this annotation?
##
flink-ml-lib/src/main/java/org/apache/flink/ml/common/util/QuantileSummary.java:
##
@@ -0,0 +1,400 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in
[jira] [Comment Edited] (FLINK-29711) Topic notification not present in metadata after 60000 ms.
[
https://issues.apache.org/jira/browse/FLINK-29711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624881#comment-17624881
]
Durgesh Mishra edited comment on FLINK-29711 at 10/28/22 5:25 AM:
--
[~mason6345] I added logs in class which implements
KafkaRecordSerializationSchema.
was (Author: JIRAUSER297316):
[~mason6345] I added logs in class which implements
KafkaRecordSerializationSchema.
> Topic notification not present in metadata after 6 ms.
> --
>
> Key: FLINK-29711
> URL: https://issues.apache.org/jira/browse/FLINK-29711
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.4, 1.14.6
>Reporter: Durgesh Mishra
>Priority: Major
>
> Failed to send data to Kafka null with
> FlinkKafkaInternalProducer\{transactionalId='null', inTransaction=false,
> closed=false}
> at
> org.apache.flink.connector.kafka.sink.KafkaWriter$WriterCallback.throwException(KafkaWriter.java:405)
> at
> org.apache.flink.connector.kafka.sink.KafkaWriter$WriterCallback.lambda$onCompletion$0(KafkaWriter.java:391)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:353)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:317)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:201)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> at java.base/java.lang.Thread.run(Unknown Source)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] liuml07 commented on pull request #20917: [hotfix][doc] Refine code examples in data_stream_api
liuml07 commented on PR #20917: URL: https://github.com/apache/flink/pull/20917#issuecomment-1294473427 Thank you @Myasuka for prompt review and commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liuml07 commented on a diff in pull request #20935: [hotfix][doc] Make referencing custom image clearer for Docker
liuml07 commented on code in PR #20935:
URL: https://github.com/apache/flink/pull/20935#discussion_r1007632470
##
docs/content.zh/docs/deployment/resource-providers/standalone/docker.md:
##
@@ -406,15 +406,15 @@ services:
You can then start creating tables and queries those.
* Note, that all required dependencies (e.g. for connectors) need to be
available in the cluster as well as the client.
- For example, if you would like to use the Kafka Connector create a custom
image with the following Dockerfile
+ For example, if you would like to use the Kafka Connector, you can build a
custom image.
+ Create a Dockerfile as follows:
```Dockerfile
FROM flink:{{< stable >}}{{< version >}}-scala{{< scala_version >}}{{<
/stable >}}{{< unstable >}}latest{{< /unstable >}}
RUN wget -P /opt/flink/lib
https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/{{<
version >}}/flink-sql-connector-kafka_scala{{< scala_version >}}-{{< version
>}}.jar
```
- and reference it (e.g via the `build`) command in the Dockerfile.
- and reference it (e.g via the `build`) command in the Dockerfile.
+ and reference this Dockerfile with the `build` command in the
`docker-compose.yml` file.
Review Comment:
Updated the doc. After building the doc screenshot is like:
https://user-images.githubusercontent.com/159186/198509170-969dc74b-4df4-4a80-a2ca-d0fa6be7dca1.png;>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] liuml07 commented on a diff in pull request #20935: [hotfix][doc] Make referencing custom image clearer for Docker
liuml07 commented on code in PR #20935:
URL: https://github.com/apache/flink/pull/20935#discussion_r1007632470
##
docs/content.zh/docs/deployment/resource-providers/standalone/docker.md:
##
@@ -406,15 +406,15 @@ services:
You can then start creating tables and queries those.
* Note, that all required dependencies (e.g. for connectors) need to be
available in the cluster as well as the client.
- For example, if you would like to use the Kafka Connector create a custom
image with the following Dockerfile
+ For example, if you would like to use the Kafka Connector, you can build a
custom image.
+ Create a Dockerfile as follows:
```Dockerfile
FROM flink:{{< stable >}}{{< version >}}-scala{{< scala_version >}}{{<
/stable >}}{{< unstable >}}latest{{< /unstable >}}
RUN wget -P /opt/flink/lib
https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/{{<
version >}}/flink-sql-connector-kafka_scala{{< scala_version >}}-{{< version
>}}.jar
```
- and reference it (e.g via the `build`) command in the Dockerfile.
- and reference it (e.g via the `build`) command in the Dockerfile.
+ and reference this Dockerfile with the `build` command in the
`docker-compose.yml` file.
Review Comment:
Updated the doc. After building the doc screenshot is like:
https://user-images.githubusercontent.com/159186/198507938-d68b6ee6-c642-48cc-8baa-5f125c210109.png;>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21183: [FLINK-29789][tests] Fix flaky test in CheckpointCoordinatorTest
flinkbot commented on PR #21183: URL: https://github.com/apache/flink/pull/21183#issuecomment-1294459420 ## CI report: * 6cb844d32692779b408f5f216899183ff2f849b7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sopan98 opened a new pull request, #21183: [FLINK-29789][tests] Fix flaky test in CheckpointCoordinatorTest
sopan98 opened a new pull request, #21183: URL: https://github.com/apache/flink/pull/21183 This PR aims to solve the issue presented here: https://issues.apache.org/jira/browse/FLINK-29789 ## What is the purpose of the change The fix is to change the HashMap of PendingCheckpoint in CheckpointCoordinator to LinkedHashMap to make the tests more stable (less flaky). ## Brief change log - Changing the map implementation of PendingCheckpoint in CheckpointCoordinator ## Verifying this change Existing tests already cover this change, and it can pass them successfully. The tests are: 1. org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex ## Does this pull request potentially affect one of the following parts: * Dependencies (does it add or upgrade a dependency): No * The public API, i.e., is any changed class annotated with `@Public(Evolving)`: No * The serializers: No * The runtime per-record code paths (performance sensitive): No * Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos,, ZooKeeper: No * The S3 file system connector: No ## Documentation Does this pull request introduce a new feature? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29789) Fix flaky tests in CheckpointCoordinatorTest
[ https://issues.apache.org/jira/browse/FLINK-29789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sopan Phaltankar updated FLINK-29789: - Description: The test org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex is flaky and has the following failure: Failures: [ERROR] Failures: [ERROR] CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex:1054 expected:<2> but was:<1> I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to find this flaky test. Command: mvn -pl flink-runtime edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest=org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest#testTriggerAndDeclineCheckpointComplex I analyzed the assertion failure and found that checkpoint1Id and checkpoint2Id are getting assigned by iterating over a HashMap. As we know, iterator() returns elements in a random order [(JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--]) and this might cause test failures for some orders. Therefore, to remove this non-determinism, we would change HashMap to LinkedHashMap. On further analysis, it was found that the Map is getting initialized on line 1894 of org.apache.flink.runtime.checkpoint.CheckpointCoordinator class. After changing from HashMap to LinkedHashMap, the above test is passing without any non-determinism. was: The test org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex is flaky and has the following failure: Failures: [ERROR] Failures: [ERROR] CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex:1054 expected:<2> but was:<1> I used the tool (NonDex|https://github.com/TestingResearchIllinois/NonDex) to find this flaky test. Command: mvn -pl flink-runtime edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest=org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest#testTriggerAndDeclineCheckpointComplex I analyzed the assertion failure and found that checkpoint1Id and checkpoint2Id are getting assigned by iterating over a HashMap. As we know, iterator() returns elements in a random order (JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--) and this might cause test failures for some orders. Therefore, to remove this non-determinism, we would change HashMap to LinkedHashMap. On further analysis, it was found that the Map is getting initialized on line 1894 of org.apache.flink.runtime.checkpoint.CheckpointCoordinator class. After changing from HashMap to LinkedHashMap, the above test is passing without any non-determinism. > Fix flaky tests in CheckpointCoordinatorTest > > > Key: FLINK-29789 > URL: https://issues.apache.org/jira/browse/FLINK-29789 > Project: Flink > Issue Type: Bug >Reporter: Sopan Phaltankar >Priority: Minor > Labels: pull-request-available > > The test > org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex > is flaky and has the following failure: > Failures: > [ERROR] Failures: > [ERROR] > CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex:1054 > expected:<2> but was:<1> > I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to > find this flaky test. > Command: mvn -pl flink-runtime edu.illinois:nondex-maven-plugun:1.1.2:nondex > -Dtest=org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest#testTriggerAndDeclineCheckpointComplex > I analyzed the assertion failure and found that checkpoint1Id and > checkpoint2Id are getting assigned by iterating over a HashMap. > As we know, iterator() returns elements in a random order > [(JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--]) > and this might cause test failures for some orders. > Therefore, to remove this non-determinism, we would change HashMap to > LinkedHashMap. > On further analysis, it was found that the Map is getting initialized on line > 1894 of org.apache.flink.runtime.checkpoint.CheckpointCoordinator class. > After changing from HashMap to LinkedHashMap, the above test is passing > without any non-determinism. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-web] godfreyhe closed pull request #574: Announcement blogpost for the 1.16 release
godfreyhe closed pull request #574: Announcement blogpost for the 1.16 release URL: https://github.com/apache/flink-web/pull/574 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] godfreyhe commented on pull request #574: Announcement blogpost for the 1.16 release
godfreyhe commented on PR #574: URL: https://github.com/apache/flink-web/pull/574#issuecomment-1294451826 merge & close -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29789) Fix flaky tests in CheckpointCoordinatorTest
[ https://issues.apache.org/jira/browse/FLINK-29789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sopan Phaltankar updated FLINK-29789: - Description: The test org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex is flaky and has the following failure: Failures: [ERROR] Failures: [ERROR] CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex:1054 expected:<2> but was:<1> I used the tool (NonDex|https://github.com/TestingResearchIllinois/NonDex) to find this flaky test. Command: mvn -pl flink-runtime edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest=org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest#testTriggerAndDeclineCheckpointComplex I analyzed the assertion failure and found that checkpoint1Id and checkpoint2Id are getting assigned by iterating over a HashMap. As we know, iterator() returns elements in a random order (JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--) and this might cause test failures for some orders. Therefore, to remove this non-determinism, we would change HashMap to LinkedHashMap. On further analysis, it was found that the Map is getting initialized on line 1894 of org.apache.flink.runtime.checkpoint.CheckpointCoordinator class. After changing from HashMap to LinkedHashMap, the above test is passing without any non-determinism. was: The test _org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport_ has the following failure: Failures: [ERROR] CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport:74 Wrong Side Output: arrays first differed at element [0]; expected:6> but was:5> I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to find this flaky test. Command: mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest='Fully Qualified Test Name' I analyzed the assertion failure and found that the root cause is because the test method calls ctx.getBroadcastState(STATE_DESCRIPTOR).immutableEntries() which calls the entrySet() method of the underlying HashMap. entrySet() returns the entries in a non-deterministic way, causing the test to be flaky. The fix would be to change _HashMap_ to _LinkedHashMap_ where the Map is getting initialized. On further analysis, it was found that the Map is getting initialized on line 53 of org.apache.flink.runtime.state.HeapBroadcastState class. After changing from HashMap to LinkedHashMap, the above test is passing. Edit: Upon making this change and running the CI, it was found that the tests org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchKeyedBroadcastExecution and org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchBroadcastExecution were failing. Upon further investigation, I found that these tests were also flaky and depended on the earlier made change. > Fix flaky tests in CheckpointCoordinatorTest > > > Key: FLINK-29789 > URL: https://issues.apache.org/jira/browse/FLINK-29789 > Project: Flink > Issue Type: Bug >Reporter: Sopan Phaltankar >Priority: Minor > Labels: pull-request-available > > The test > org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex > is flaky and has the following failure: > Failures: > [ERROR] Failures: > [ERROR] > CheckpointCoordinatorTest.testTriggerAndDeclineCheckpointComplex:1054 > expected:<2> but was:<1> > I used the tool (NonDex|https://github.com/TestingResearchIllinois/NonDex) to > find this flaky test. > Command: mvn -pl flink-runtime edu.illinois:nondex-maven-plugun:1.1.2:nondex > -Dtest=org.apache.flink.runtime.checkpoint.CheckpointCoordinatorTest#testTriggerAndDeclineCheckpointComplex > I analyzed the assertion failure and found that checkpoint1Id and > checkpoint2Id are getting assigned by iterating over a HashMap. > As we know, iterator() returns elements in a random order > (JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--) > and this might cause test failures for some orders. > Therefore, to remove this non-determinism, we would change HashMap to > LinkedHashMap. > On further analysis, it was found that the Map is getting initialized on line > 1894 of org.apache.flink.runtime.checkpoint.CheckpointCoordinator class. > After changing from HashMap to LinkedHashMap, the above test is passing > without any non-determinism. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29789) Fix flaky tests in CheckpointCoordinatorTest
Sopan Phaltankar created FLINK-29789: Summary: Fix flaky tests in CheckpointCoordinatorTest Key: FLINK-29789 URL: https://issues.apache.org/jira/browse/FLINK-29789 Project: Flink Issue Type: Bug Reporter: Sopan Phaltankar The test _org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport_ has the following failure: Failures: [ERROR] CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport:74 Wrong Side Output: arrays first differed at element [0]; expected:6> but was:5> I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to find this flaky test. Command: mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest='Fully Qualified Test Name' I analyzed the assertion failure and found that the root cause is because the test method calls ctx.getBroadcastState(STATE_DESCRIPTOR).immutableEntries() which calls the entrySet() method of the underlying HashMap. entrySet() returns the entries in a non-deterministic way, causing the test to be flaky. The fix would be to change _HashMap_ to _LinkedHashMap_ where the Map is getting initialized. On further analysis, it was found that the Map is getting initialized on line 53 of org.apache.flink.runtime.state.HeapBroadcastState class. After changing from HashMap to LinkedHashMap, the above test is passing. Edit: Upon making this change and running the CI, it was found that the tests org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchKeyedBroadcastExecution and org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchBroadcastExecution were failing. Upon further investigation, I found that these tests were also flaky and depended on the earlier made change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29611) Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest
[ https://issues.apache.org/jira/browse/FLINK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619810#comment-17619810 ] Sopan Phaltankar edited comment on FLINK-29611 at 10/28/22 4:35 AM: [~martijnvisser] This test depends on the order of iteration in `HashMap.entrySet()` and can fail for some orders. I am running this on my local machine. I have used a maven plugin NonDex, which can be used to identify such tests. One can reproduce it with the command `mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest=org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest#testMultiStateSupport`. Even if the test was not failing during the daily jobs, it'd be good to not depend on the `HashMap.entrySet()` that gives back results in an undefined manner [JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--] for this. Therefore, to remove non-determinism completely, we can do this change. was (Author: JIRAUSER296865): [~martijnvisser] This test depends on the order of iteration in `HashMap.entrySet()` and can fail for some orders. I am running this on my local machine. I have used a maven plugin NonDex, which can be used to identify such tests. One can reproduce it with the command `mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest=org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest#testMultiStateSupport`. Even if the test was not failing during the daily jobs, it'd be good to not depend on the `HashMap.entrySet()` that gives back results in an undefined manner [JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--]for this. Therefore, to remove non-determinism completely, we can do this change. > Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest > -- > > Key: FLINK-29611 > URL: https://issues.apache.org/jira/browse/FLINK-29611 > Project: Flink > Issue Type: Bug >Reporter: Sopan Phaltankar >Priority: Minor > Labels: pull-request-available > > The test > _org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport_ > has the following failure: > Failures: > [ERROR] CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport:74 > Wrong Side Output: arrays first differed at element [0]; expected: 15 : 9:key.6->6> but was:5> > I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to > find this flaky test. > Command: mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest='Fully > Qualified Test Name' > I analyzed the assertion failure and found that the root cause is because the > test method calls ctx.getBroadcastState(STATE_DESCRIPTOR).immutableEntries() > which calls the entrySet() method of the underlying HashMap. entrySet() > returns the entries in a non-deterministic way, causing the test to be flaky. > The fix would be to change _HashMap_ to _LinkedHashMap_ where the Map is > getting initialized. > On further analysis, it was found that the Map is getting initialized on line > 53 of org.apache.flink.runtime.state.HeapBroadcastState class. > After changing from HashMap to LinkedHashMap, the above test is passing. > Edit: Upon making this change and running the CI, it was found that the tests > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchKeyedBroadcastExecution > and > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchBroadcastExecution > were failing. Upon further investigation, I found that these tests were also > flaky and depended on the earlier made change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29611) Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest
[ https://issues.apache.org/jira/browse/FLINK-29611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17619810#comment-17619810 ] Sopan Phaltankar edited comment on FLINK-29611 at 10/28/22 4:34 AM: [~martijnvisser] This test depends on the order of iteration in `HashMap.entrySet()` and can fail for some orders. I am running this on my local machine. I have used a maven plugin NonDex, which can be used to identify such tests. One can reproduce it with the command `mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest=org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest#testMultiStateSupport`. Even if the test was not failing during the daily jobs, it'd be good to not depend on the `HashMap.entrySet()` that gives back results in an undefined manner [JavaDoc|https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--]for this. Therefore, to remove non-determinism completely, we can do this change. was (Author: JIRAUSER296865): [~martijnvisser] This test depends on the order of iteration in `HashMap.entrySet()` and can fail for some orders. I am running this on my local machine. I have used a maven plugin NonDex, which can be used to identify such tests. One can reproduce it with the command `mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest=org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest#testMultiStateSupport`. Even if the test was not failing during the daily jobs, it'd be good to not depend on the `HashMap.entrySet()` that gives back results in an undefined manner [JavaDoc|[https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#entrySet--]] for this. Therefore, to remove non-determinism completely, we can do this change. > Fix flaky tests in CoBroadcastWithNonKeyedOperatorTest > -- > > Key: FLINK-29611 > URL: https://issues.apache.org/jira/browse/FLINK-29611 > Project: Flink > Issue Type: Bug >Reporter: Sopan Phaltankar >Priority: Minor > Labels: pull-request-available > > The test > _org.apache.flink.streaming.api.operators.co.CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport_ > has the following failure: > Failures: > [ERROR] CoBroadcastWithNonKeyedOperatorTest.testMultiStateSupport:74 > Wrong Side Output: arrays first differed at element [0]; expected: 15 : 9:key.6->6> but was:5> > I used the tool [NonDex|https://github.com/TestingResearchIllinois/NonDex] to > find this flaky test. > Command: mvn edu.illinois:nondex-maven-plugun:1.1.2:nondex -Dtest='Fully > Qualified Test Name' > I analyzed the assertion failure and found that the root cause is because the > test method calls ctx.getBroadcastState(STATE_DESCRIPTOR).immutableEntries() > which calls the entrySet() method of the underlying HashMap. entrySet() > returns the entries in a non-deterministic way, causing the test to be flaky. > The fix would be to change _HashMap_ to _LinkedHashMap_ where the Map is > getting initialized. > On further analysis, it was found that the Map is getting initialized on line > 53 of org.apache.flink.runtime.state.HeapBroadcastState class. > After changing from HashMap to LinkedHashMap, the above test is passing. > Edit: Upon making this change and running the CI, it was found that the tests > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchKeyedBroadcastExecution > and > org.apache.flink.api.datastream.DataStreamBatchExecutionITCase.batchBroadcastExecution > were failing. Upon further investigation, I found that these tests were also > flaky and depended on the earlier made change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] reswqa commented on a diff in pull request #21137: [FLINK-29234][runtime] JobMasterServiceLeadershipRunner handle leader event in a separate executor to avoid dead lock
reswqa commented on code in PR #21137:
URL: https://github.com/apache/flink/pull/21137#discussion_r100709
##
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java:
##
@@ -683,21 +665,66 @@ public void testJobAlreadyDone() throws Exception {
jobManagerRunner.getResultFuture();
JobManagerRunnerResult result = resultFuture.get();
-assertEquals(
-JobStatus.FAILED,
-
result.getExecutionGraphInfo().getArchivedExecutionGraph().getState());
+
assertThat(result.getExecutionGraphInfo().getArchivedExecutionGraph().getState())
+.isEqualTo(JobStatus.FAILED);
}
}
+@Test
+void
testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip()
+throws Exception {
+final TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory
+testingLeaderElectionDriverFactory =
+new
TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory();
+final LeaderElectionService defaultLeaderElectionService =
+new
DefaultLeaderElectionService(testingLeaderElectionDriverFactory);
+
+final JobMasterServiceLeadershipRunner jobManagerRunner =
+newJobMasterServiceLeadershipRunnerBuilder()
+.setJobMasterServiceProcessFactory(
+
TestingJobMasterServiceProcessFactory.newBuilder().build())
+.setLeaderElectionService(defaultLeaderElectionService)
+.build();
+
+jobManagerRunner.start();
+final TestingLeaderElectionDriver currentLeaderDriver =
+Preconditions.checkNotNull(
+
testingLeaderElectionDriverFactory.getCurrentLeaderDriver());
+
+final CheckedThread contenderCloseThread =
+new CheckedThread() {
+@Override
+public void go() {
+try {
+jobManagerRunner.close();
+} catch (Exception e) {
+throw new RuntimeException(e);
+}
+}
+};
+contenderCloseThread.start();
+
+// grant leadership.
+currentLeaderDriver.isLeader();
Review Comment:
@XComp Thank you for your advice.
At the beginning, I didn't think of a good way to control the completion of
completableFuture, and the probability of deadlock reproduce in my local
environment is very high. But your suggestion did find a good control injection
point, so i decided to test it in this way.
But if we want to reproduce the problem 100%, it seems that we need another
control point. Maybe `TestingLeaderElectionDriver#isLeader` is a suitable
choice. What's your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625417#comment-17625417 ] Hangxiang Yu commented on FLINK-29788: -- Sure. It's known limitation that changelog hasn't supported native savepoint in 1.16. So I think we could just disable changelog when triggering native savepoint in this case. It will not block the release of 1.16. BTW, IIUC, Should the migration it case be executed before we decide to release 1.16 ? [~hxbks2ks] > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Critical > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29565) In Flink per job mode, the logs printed by taskManager on the web UI will not be highlighted, because the log contents are annotated due to special symbols, which will
[ https://issues.apache.org/jira/browse/FLINK-29565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625409#comment-17625409 ] wangshiwei commented on FLINK-29565: yes > In Flink per job mode, the logs printed by taskManager on the web UI will not > be highlighted, because the log contents are annotated due to special > symbols, which will affect the use experience. For more information, see Fig > > > Key: FLINK-29565 > URL: https://issues.apache.org/jira/browse/FLINK-29565 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.14.3 >Reporter: wangshiwei >Priority: Minor > Attachments: image-2022-10-10-18-40-27-721.png, > image-2022-10-10-18-43-31-897.png, image-2022-10-10-18-43-53-713.png, > image-2022-10-10-18-45-17-228.png, image-2022-10-10-19-02-29-796.png, > image-2022-10-10-19-03-27-670.png > > > > !image-2022-10-10-19-03-27-670.png|width=580,height=317! > !image-2022-10-10-18-43-53-713.png|width=726,height=47! > This kind of '/*' content will appear in the print log of the > logEnvironmentInfo method in the EnvironmentInformation class. The following > logs will be commented out without highlighting > *verification* > !image-2022-10-10-18-45-17-228.png|width=880,height=161! > After manually printing '*/' in the business code, the log is normal > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] liyubin117 commented on pull request #21182: [FLINK-29679][table] DESCRIBE statement shows column comment
liyubin117 commented on PR #21182: URL: https://github.com/apache/flink/pull/21182#issuecomment-1294406889 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on a diff in pull request #162: [FLINK-29593] Add QuantileSummary to help calculate approximate quant…
zhipeng93 commented on code in PR #162:
URL: https://github.com/apache/flink-ml/pull/162#discussion_r1007572419
##
flink-ml-lib/src/main/java/org/apache/flink/ml/common/util/QuantileSummary.java:
##
@@ -0,0 +1,400 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Helper class to compute an approximate quantile summary. This
implementation is based on the
+ * algorithm proposed in the paper: "Space-efficient Online Computation of
Quantile Summaries" by
+ * Greenwald, Michael and Khanna, Sanjeev.
(https://doi.org/10.1145/375663.375670)
+ */
+@Internal
+public class QuantileSummary implements Serializable {
+
+/** The target relative error. */
+private final double relativeError;
+
+/**
+ * The compression threshold. After the internal buffer of statistics
crosses this size, it
+ * attempts to compress the statistics together.
+ */
+private final int compressThreshold;
+
+/** The count of all the elements inserted to be calculated. */
+private final long count;
+
+/** A buffer of quantile statistics. */
+private final List sampled;
+
+/** The default size of head buffer. */
+private static final int DEFAULT_HEAD_SIZE = 5;
+
+/** The default compression threshold. */
+private static final int DEFAULT_COMPRESS_THRESHOLD = 1;
+
+/** A buffer of the latest samples seen so far. */
+private List headBuffer = new ArrayList<>(DEFAULT_HEAD_SIZE);
Review Comment:
nit: Do you think making it a `double[]` could be more memory efficient? I
am also fine the keep the current implementation.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/common/util/QuantileSummary.java:
##
@@ -0,0 +1,400 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.util;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import java.util.stream.IntStream;
+
+/**
+ * Helper class to compute an approximate quantile summary. This
implementation is based on the
+ * algorithm proposed in the paper: "Space-efficient Online Computation of
Quantile Summaries" by
+ * Greenwald, Michael and Khanna, Sanjeev.
(https://doi.org/10.1145/375663.375670)
+ */
+@Internal
+public class QuantileSummary implements Serializable {
+
+/** The target relative error. */
+private final double relativeError;
+
+/**
+ * The compression threshold. After the internal buffer of statistics
crosses this size, it
+ * attempts to compress the statistics together.
+ */
+private final int compressThreshold;
+
+/** The count of all the elements inserted to be calculated. */
+private final long count;
+
+/** A buffer of quantile
[jira] [Updated] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huang Xingbo updated FLINK-29788: - Priority: Critical (was: Major) > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Critical > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
[ https://issues.apache.org/jira/browse/FLINK-29788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625404#comment-17625404 ] Huang Xingbo commented on FLINK-29788: -- [~masteryhx] Could you help take a look? Thx. > StatefulJobWBroadcastStateMigrationITCase failed in native savepoints > - > > Key: FLINK-29788 > URL: https://issues.apache.org/jira/browse/FLINK-29788 > Project: Flink > Issue Type: Bug > Components: Release System, Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Huang Xingbo >Priority: Major > Attachments: image-2022-10-28-11-18-45-471.png > > > !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] liyubin117 commented on pull request #21182: [FLINK-29679][table] DESCRIBE statement shows column comment
liyubin117 commented on PR #21182: URL: https://github.com/apache/flink/pull/21182#issuecomment-1294397909 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29788) StatefulJobWBroadcastStateMigrationITCase failed in native savepoints
Huang Xingbo created FLINK-29788: Summary: StatefulJobWBroadcastStateMigrationITCase failed in native savepoints Key: FLINK-29788 URL: https://issues.apache.org/jira/browse/FLINK-29788 Project: Flink Issue Type: Bug Components: Release System, Runtime / State Backends Affects Versions: 1.16.0 Reporter: Huang Xingbo Attachments: image-2022-10-28-11-18-45-471.png !image-2022-10-28-11-18-45-471.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] xintongsong commented on pull request #21122: [FLINK-28889] Hybrid shuffle supports multiple consumer and broadcast optimization
xintongsong commented on PR #21122: URL: https://github.com/apache/flink/pull/21122#issuecomment-1294395156 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28863) Snapshot result of RocksDB native savepoint should have empty shared-state
[ https://issues.apache.org/jira/browse/FLINK-28863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625402#comment-17625402 ] Jinzhong Li commented on FLINK-28863: - In my opinion, for Rocksdb native savepoint SnapshotResult, we should put [sstFiles|https://github.com/apache/flink/blob/bb9f2525e6e16d00ef2f0739d9cb96c2e47e35e7/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L302] into privateState Map of IncrementalRemoteKeyedStateHandle to fix this issue. This change has no effect on restore, which [downloads both the priavateStates and shareStates|[https://github.com/apache/flink/blob/35c5f674041bcefea93e1de459cea0d1789f98e0/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateDownloader.java#L53].] [~yunta] WDYT? If my understanding is correct, I can fix the issue. > Snapshot result of RocksDB native savepoint should have empty shared-state > -- > > Key: FLINK-28863 > URL: https://issues.apache.org/jira/browse/FLINK-28863 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: Yun Tang >Priority: Major > Fix For: 1.17.0, 1.15.3, 1.16.1 > > > The current snapshot result of RocksDB native savepoint has non-empty shared > state, which is obviously not correct as all snapshot artifacts already stay > in the exclusive checkpoint scope folder. > This does not bring real harmful result due to we would not register the > snapshot results of RocksDB native savepoint. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29787) fix ci METHOD_NEW_DEFAULT issue
Yubin Li created FLINK-29787: Summary: fix ci METHOD_NEW_DEFAULT issue Key: FLINK-29787 URL: https://issues.apache.org/jira/browse/FLINK-29787 Project: Flink Issue Type: Bug Components: Build System / CI Reporter: Yubin Li `org.apache.flink.api.connector.source.SourceReader` declared a new default function `pauseOrResumeSplits()`, japicmp plugin failed during ci running, we should configure the plugin to make it compatible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Myasuka merged pull request #20917: [hotfix][doc] Refine code examples in data_stream_api
Myasuka merged PR #20917: URL: https://github.com/apache/flink/pull/20917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28973) Extending /jars/:jarid/plan API to support setting Flink configs
[ https://issues.apache.org/jira/browse/FLINK-28973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625399#comment-17625399 ] Zhanghao Chen commented on FLINK-28973: --- Hi [~gaoyunhaii]. The concern is totally valid, but as have been discussed in [FLIP-256: Support Job Dynamic Parameter With Flink Rest Api - Apache Flink - Apache Software Foundation|https://cwiki.apache.org/confluence/display/FLINK/FLIP-256%3A+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api], a pragmatic approach would just be allow to take both types of config options as input, and simply ignore the cluster-level ones, which is also consistent with how we handle it when submitting jobs. > Extending /jars/:jarid/plan API to support setting Flink configs > > > Key: FLINK-28973 > URL: https://issues.apache.org/jira/browse/FLINK-28973 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: Zhanghao Chen >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29565) In Flink per job mode, the logs printed by taskManager on the web UI will not be highlighted, because the log contents are annotated due to special symbols, which will
[ https://issues.apache.org/jira/browse/FLINK-29565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625396#comment-17625396 ] Yun Gao commented on FLINK-29565: - Hi [~1336399775] then would you like to open a PR to fix this issue? > In Flink per job mode, the logs printed by taskManager on the web UI will not > be highlighted, because the log contents are annotated due to special > symbols, which will affect the use experience. For more information, see Fig > > > Key: FLINK-29565 > URL: https://issues.apache.org/jira/browse/FLINK-29565 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.14.3 >Reporter: wangshiwei >Priority: Minor > Attachments: image-2022-10-10-18-40-27-721.png, > image-2022-10-10-18-43-31-897.png, image-2022-10-10-18-43-53-713.png, > image-2022-10-10-18-45-17-228.png, image-2022-10-10-19-02-29-796.png, > image-2022-10-10-19-03-27-670.png > > > > !image-2022-10-10-19-03-27-670.png|width=580,height=317! > !image-2022-10-10-18-43-53-713.png|width=726,height=47! > This kind of '/*' content will appear in the print log of the > logEnvironmentInfo method in the EnvironmentInformation class. The following > logs will be commented out without highlighting > *verification* > !image-2022-10-10-18-45-17-228.png|width=880,height=161! > After manually printing '*/' in the business code, the log is normal > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] HuangXingBo commented on pull request #20859: [release] release notes for the 1.16 release
HuangXingBo commented on PR #20859: URL: https://github.com/apache/flink/pull/20859#issuecomment-1294382816 @zentol Ok. I have addressed the comment at the latest commit. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21182: [FLINK-29679][table] DESCRIBE statement shows column comment
flinkbot commented on PR #21182: URL: https://github.com/apache/flink/pull/21182#issuecomment-1294381431 ## CI report: * 0a7afa1d3cb5370c44b1bea729afa6577ae7d3c9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29565) In Flink per job mode, the logs printed by taskManager on the web UI will not be highlighted, because the log contents are annotated due to special symbols, which will
[ https://issues.apache.org/jira/browse/FLINK-29565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625395#comment-17625395 ] wangshiwei commented on FLINK-29565: yes, I have > In Flink per job mode, the logs printed by taskManager on the web UI will not > be highlighted, because the log contents are annotated due to special > symbols, which will affect the use experience. For more information, see Fig > > > Key: FLINK-29565 > URL: https://issues.apache.org/jira/browse/FLINK-29565 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.14.3 >Reporter: wangshiwei >Priority: Minor > Attachments: image-2022-10-10-18-40-27-721.png, > image-2022-10-10-18-43-31-897.png, image-2022-10-10-18-43-53-713.png, > image-2022-10-10-18-45-17-228.png, image-2022-10-10-19-02-29-796.png, > image-2022-10-10-19-03-27-670.png > > > > !image-2022-10-10-19-03-27-670.png|width=580,height=317! > !image-2022-10-10-18-43-53-713.png|width=726,height=47! > This kind of '/*' content will appear in the print log of the > logEnvironmentInfo method in the EnvironmentInformation class. The following > logs will be commented out without highlighting > *verification* > !image-2022-10-10-18-45-17-228.png|width=880,height=161! > After manually printing '*/' in the business code, the log is normal > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] reswqa commented on a diff in pull request #21137: [FLINK-29234][runtime] JobMasterServiceLeadershipRunner handle leader event in a separate executor to avoid dead lock
reswqa commented on code in PR #21137:
URL: https://github.com/apache/flink/pull/21137#discussion_r1007555607
##
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java:
##
@@ -683,21 +665,66 @@ public void testJobAlreadyDone() throws Exception {
jobManagerRunner.getResultFuture();
JobManagerRunnerResult result = resultFuture.get();
-assertEquals(
-JobStatus.FAILED,
-
result.getExecutionGraphInfo().getArchivedExecutionGraph().getState());
+
assertThat(result.getExecutionGraphInfo().getArchivedExecutionGraph().getState())
+.isEqualTo(JobStatus.FAILED);
}
}
+@Test
+void
testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip()
+throws Exception {
+final TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory
+testingLeaderElectionDriverFactory =
+new
TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory();
+final LeaderElectionService defaultLeaderElectionService =
+new
DefaultLeaderElectionService(testingLeaderElectionDriverFactory);
+
+final JobMasterServiceLeadershipRunner jobManagerRunner =
+newJobMasterServiceLeadershipRunnerBuilder()
+.setJobMasterServiceProcessFactory(
+
TestingJobMasterServiceProcessFactory.newBuilder().build())
+.setLeaderElectionService(defaultLeaderElectionService)
+.build();
+
+jobManagerRunner.start();
+final TestingLeaderElectionDriver currentLeaderDriver =
+Preconditions.checkNotNull(
+
testingLeaderElectionDriverFactory.getCurrentLeaderDriver());
+
+final CheckedThread contenderCloseThread =
+new CheckedThread() {
+@Override
+public void go() {
+try {
+jobManagerRunner.close();
+} catch (Exception e) {
+throw new RuntimeException(e);
+}
+}
+};
+contenderCloseThread.start();
+
+// grant leadership.
+currentLeaderDriver.isLeader();
Review Comment:
@XComp Thank you for your advice.
At the beginning, I didn't think of a good way to control the completion of
completableFuture, and the probability of deadlock reproduce in my local
environment is very high. But your suggestion did find a good control injection
point, so i decided to test it in this way.
But if we want to reproduce the problem 100%, it seems that we need another
control point. Maybe `TestingLeaderElectionDriver#isLeader` is a suitable
choice. What's your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] reswqa commented on a diff in pull request #21137: [FLINK-29234][runtime] JobMasterServiceLeadershipRunner handle leader event in a separate executor to avoid dead lock
reswqa commented on code in PR #21137:
URL: https://github.com/apache/flink/pull/21137#discussion_r1007555607
##
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java:
##
@@ -683,21 +665,66 @@ public void testJobAlreadyDone() throws Exception {
jobManagerRunner.getResultFuture();
JobManagerRunnerResult result = resultFuture.get();
-assertEquals(
-JobStatus.FAILED,
-
result.getExecutionGraphInfo().getArchivedExecutionGraph().getState());
+
assertThat(result.getExecutionGraphInfo().getArchivedExecutionGraph().getState())
+.isEqualTo(JobStatus.FAILED);
}
}
+@Test
+void
testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip()
+throws Exception {
+final TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory
+testingLeaderElectionDriverFactory =
+new
TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory();
+final LeaderElectionService defaultLeaderElectionService =
+new
DefaultLeaderElectionService(testingLeaderElectionDriverFactory);
+
+final JobMasterServiceLeadershipRunner jobManagerRunner =
+newJobMasterServiceLeadershipRunnerBuilder()
+.setJobMasterServiceProcessFactory(
+
TestingJobMasterServiceProcessFactory.newBuilder().build())
+.setLeaderElectionService(defaultLeaderElectionService)
+.build();
+
+jobManagerRunner.start();
+final TestingLeaderElectionDriver currentLeaderDriver =
+Preconditions.checkNotNull(
+
testingLeaderElectionDriverFactory.getCurrentLeaderDriver());
+
+final CheckedThread contenderCloseThread =
+new CheckedThread() {
+@Override
+public void go() {
+try {
+jobManagerRunner.close();
+} catch (Exception e) {
+throw new RuntimeException(e);
+}
+}
+};
+contenderCloseThread.start();
+
+// grant leadership.
+currentLeaderDriver.isLeader();
Review Comment:
@XComp Thank you for your advice.
At the beginning, I didn't think of a good way to control the completion of
completableFuture, and the probability of deadlock reproduce in my local
environment is very high. But your suggestion did find a good control injection
point, so i decided to test it in this way.
But if we want to reproduce the problem 100%, it seems that we need another
control point. Maybe TestingLeaderElectionDriver#isLeader is a suitable choice.
What's your opinion?
##
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunnerTest.java:
##
@@ -683,21 +665,66 @@ public void testJobAlreadyDone() throws Exception {
jobManagerRunner.getResultFuture();
JobManagerRunnerResult result = resultFuture.get();
-assertEquals(
-JobStatus.FAILED,
-
result.getExecutionGraphInfo().getArchivedExecutionGraph().getState());
+
assertThat(result.getExecutionGraphInfo().getArchivedExecutionGraph().getState())
+.isEqualTo(JobStatus.FAILED);
}
}
+@Test
+void
testJobMasterServiceLeadershipRunnerCloseWhenElectionServiceGrantLeaderShip()
+throws Exception {
+final TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory
+testingLeaderElectionDriverFactory =
+new
TestingLeaderElectionDriver.TestingLeaderElectionDriverFactory();
+final LeaderElectionService defaultLeaderElectionService =
+new
DefaultLeaderElectionService(testingLeaderElectionDriverFactory);
+
+final JobMasterServiceLeadershipRunner jobManagerRunner =
+newJobMasterServiceLeadershipRunnerBuilder()
+.setJobMasterServiceProcessFactory(
+
TestingJobMasterServiceProcessFactory.newBuilder().build())
+.setLeaderElectionService(defaultLeaderElectionService)
+.build();
+
+jobManagerRunner.start();
+final TestingLeaderElectionDriver currentLeaderDriver =
+Preconditions.checkNotNull(
+
testingLeaderElectionDriverFactory.getCurrentLeaderDriver());
+
+final CheckedThread contenderCloseThread =
+new CheckedThread() {
+@Override
+public void go() {
+try {
+
[GitHub] [flink] liyubin117 closed pull request #21116: [FLINK-29679][table] DESCRIBE statement shows column comment
liyubin117 closed pull request #21116: [FLINK-29679][table] DESCRIBE statement shows column comment URL: https://github.com/apache/flink/pull/21116 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liyubin117 opened a new pull request, #21182: [FLINK-29679][table] DESCRIBE statement shows column comment
liyubin117 opened a new pull request, #21182: URL: https://github.com/apache/flink/pull/21182 ## What is the purpose of the change DESCRIBE statement print column comment to make results user-friendly, such as help analysts write sql adaptive to corresponding business logics. ## Brief change log * use new schema framework to get comment. ## Verifying this change This change added tests and can be verified as follows: flink-table/flink-sql-client/src/test/resources/sql/table.q ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29765) SQL query not executing properly
[
https://issues.apache.org/jira/browse/FLINK-29765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625391#comment-17625391
]
Shengkai Fang commented on FLINK-29765:
---
It's better you can share more infos, e.g. source table schema, json plan?
> SQL query not executing properly
>
>
> Key: FLINK-29765
> URL: https://issues.apache.org/jira/browse/FLINK-29765
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.0
>Reporter: Aqib Mehmood
>Priority: Major
>
> I am using this query to compare difference between last and second last
> price of our order sku
> WITH CTE AS (
> SELECT a.sku, a.name, a.updatedAt, b.price FROM (
> SELECT sku, name, max(updatedAt) AS updatedAt from (
> SELECT sku, name, updatedAt FROM wms.PurchaseOrderProduct
> WHERE CONCAT(sku, DATE_FORMAT(updatedAt, '%Y-%m-%d %H:%m:%s'))
> not in (
> SELECT CONCAT(sku, DATE_FORMAT(updatedAt, '%Y-%m-%d
> %H:%m:%s')) FROM (
> SELECT sku, max(updatedAt) as updatedAt from
> wms.PurchaseOrderProduct
> GROUP BY sku
> ) AS x
> )
> ) AS z
> GROUP BY sku, name
> ) AS a
> LEFT JOIN wms.PurchaseOrderProduct b
> ON a.sku=b.sku AND a.name=b.name and a.updatedAt=b.updatedAt
> )
> SELECT a.sku, a.name, a.updatedAt AS latestupdatedAt, a.price AS latestPrice,
> b.updatedAt AS lastUpdatedAt, b.price AS lastPrice
> FROM (
> SELECT a.sku, a.name, a.updatedAt, b.price from (
> SELECT sku, name, max(updatedAt) as updatedAt from
> wms.PurchaseOrderProduct
> GROUP BY sku, name
> ) AS a
> LEFT JOIN wms.PurchaseOrderProduct b
> ON a.sku=b.sku AND a.name=b.name and a.updatedAt=b.updatedAt
> ) AS a
> LEFT JOIN CTE AS b
> ON a.sku=b.sku AND a.name=b.name;
> This issue is that Im getting *NULLs* for columns *lastUpdatedAt* and
> {*}lastPrice{*}. But when I run the same query on our prod database, I'm
> getting desired results.I suspect that flink is not processing the entire
> query before giving the results.
> I get desired results for a couple of rows in while *lastUpdatedAt* and
> *lastPrice* are not *NULL* in the beginning of the table{*}.{*} But then
> after that the entire two columns return *NULLs*
> I would like to know why flink is not executing the above query properly?TIA
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29748) Expose the optimize phase in the connector context
[
https://issues.apache.org/jira/browse/FLINK-29748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625388#comment-17625388
]
Shengkai Fang commented on FLINK-29748:
---
I think you can validate when StreamExecSource invokes getScanRuntimeProvider.
When execnode -> transformat, it means the optimization finishes.
> Expose the optimize phase in the connector context
> --
>
> Key: FLINK-29748
> URL: https://issues.apache.org/jira/browse/FLINK-29748
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner, Table SQL / Runtime
>Reporter: Aitozi
>Priority: Minor
>
> Currently, in the connector it can not know whether the whole optimize is
> finished.
> When the optimize finished, the all information is static, eg: the reading
> partitions. If I want to validate the final optimized result (like whether
> the reading partition is too much or empty), it needs the context of what is
> the current phase. I think the {{ScanContext}} is ok to expose this
> information.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #160: [FLINK-29434] Add AlgoOperator for Splitter
yunfengzhou-hub commented on code in PR #160:
URL: https://github.com/apache/flink-ml/pull/160#discussion_r1007543761
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/randomsplitter/RandomSplitterParams.java:
##
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.randomsplitter;
+
+import org.apache.flink.ml.param.DoubleArrayParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidator;
+import org.apache.flink.ml.param.WithParams;
+
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Set;
+
+/**
+ * Params of {@link RandomSplitter}.
+ *
+ * @param The class type of this instance.
+ */
+public interface RandomSplitterParams extends WithParams {
+Param WEIGHTS =
+new DoubleArrayParam(
+"weights", "The weights of data splitting.", null,
weightsValidator());
Review Comment:
Weight is different from fraction. Weight means the proportion of the number
of elements in each output split table, instead of a possibility threshold
like the fraction in the previous implementation.
`setWeight(1,1)` is equal to `setThreshold(0.5)`, `setWeight(1,2,2)` is
equal to `setThreshold(0.2,0.6)`.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/randomsplitter/RandomSplitter.java:
##
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.randomsplitter;
+
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.ml.api.AlgoOperator;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.OutputTag;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Random;
+
+/** An AlgoOperator which splits a datastream into N datastreams according to
the given weights. */
+public class RandomSplitter
+implements AlgoOperator,
RandomSplitterParams {
+private final Map, Object> paramMap = new HashMap<>();
+
+public RandomSplitter() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public Table[] transform(Table... inputs) {
+Preconditions.checkArgument(inputs.length == 1);
+StreamTableEnvironment tEnv =
+(StreamTableEnvironment) ((TableImpl)
inputs[0]).getTableEnvironment();
+RowTypeInfo outputTypeInfo =
TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+
+final Double[] weights = getWeights();
+OutputTag[] outputTags = new OutputTag[weights.length];
+for (int i = 0; i <
[jira] [Updated] (FLINK-29786) VarianceThresholdSelector Uses FeaturesCol Incorrectly
[ https://issues.apache.org/jira/browse/FLINK-29786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29786: --- Labels: pull-request-available (was: ) > VarianceThresholdSelector Uses FeaturesCol Incorrectly > -- > > Key: FLINK-29786 > URL: https://issues.apache.org/jira/browse/FLINK-29786 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Reporter: Jiang Xin >Priority: Major > Labels: pull-request-available > Fix For: ml-2.2.0 > > > Flink ML has both `FeaturesCol` and `InputCol` to be used to set input. The > `FeaturesCol` should only be used with `LabelCol`, while `InputCol` can be > used more generally. > The `VarianceThresholdSelector` should use `InputCol` instead of > `FeaturesCol`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] jiangxin369 opened a new pull request, #167: [FLINK-29786] VarianceThresholdSelector Uses FeaturesCol Incorrectly
jiangxin369 opened a new pull request, #167: URL: https://github.com/apache/flink-ml/pull/167 ## What is the purpose of the change Replace `FeaturesCol` with `InputCol` in VarianceThresholdSelector. ## Brief change log - Add Javadoc for `HasFeaturesCol` to distinguish `HasInputCol` - Replace `FeaturesCol` with `InputCol` in VarianceThresholdSelector. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #21122: [FLINK-28889] Hybrid shuffle supports multiple consumer and broadcast optimization
xintongsong commented on PR #21122: URL: https://github.com/apache/flink/pull/21122#issuecomment-1294359915 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 merged pull request #21179: [BP-1.16][FLINK-29539][Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
wangyang0918 merged PR #21179: URL: https://github.com/apache/flink/pull/21179 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 merged pull request #20982: [FLINK-29539] [Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
wangyang0918 merged PR #20982: URL: https://github.com/apache/flink/pull/20982 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-29572) Flink Task Manager skip loopback interface for resource manager registration
[ https://issues.apache.org/jira/browse/FLINK-29572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625383#comment-17625383 ] Xintong Song edited comment on FLINK-29572 at 10/28/22 2:15 AM: You don't have to configure different ports for each task manager. You just need to remove `taskmanager.rpc.port` from your configuration, and Flink by default should use random ports. IMHO, allowing components from different hosts to bi-directionally communicate with each other via the loopback address does not sounds like a common use case. It seems to be a violation of the loopback address semantics. This is probably more a problem of the proxy software, rather than Flink. I'd love to help you find a workaround to get things work if possible. However, as one of the maintainers of the Flink project, I'd be negative to introduce such a knob for such an unusual use case. was (Author: xintongsong): You don't have to configure different ports for each task manager. You just need to remove `taskmanager.rpc.port` from your configuration, and Flink by default should use random ports. IMHO, allowing components from different hosts to bi-directionally communicate with each other via the loopback address does not sounds like a common use case. It seems to be a violation of the loopback address semantics. This is probably more a problem of the proxy software, rather than Flink. I'd love to help you find a workaround to get things work. However, as one of the maintainers of the Flink project, I'd be negative to introduce such a knob for such an unusual use case. > Flink Task Manager skip loopback interface for resource manager registration > > > Key: FLINK-29572 > URL: https://issues.apache.org/jira/browse/FLINK-29572 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.15.2 > Environment: Flink 1.15.2 > Kubernetes with Istio Proxy >Reporter: Kevin Li >Priority: Major > > Currently Flink Task Manager use different local interface to bind to connect > to Resource Manager. First one is Loopback interface. Normally if Job Manager > is running on remote host/container, using loopback interface to connect will > fail and it will pick up correct IP address. > However, if Task Manager is running with some proxy, loopback interface can > connect to remote host as well. This will result 127.0.0.1 reported to > Resource Manager during registration, even Job Manager/Resource Manager runs > on remote host, and problem will happen. For us, only one Task Manager can > register in this case. > I suggest adding configuration to skip Loopback interface check if we know > Job/Resource Manager is running on remote host/container. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29572) Flink Task Manager skip loopback interface for resource manager registration
[ https://issues.apache.org/jira/browse/FLINK-29572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625383#comment-17625383 ] Xintong Song commented on FLINK-29572: -- You don't have to configure different ports for each task manager. You just need to remove `taskmanager.rpc.port` from your configuration, and Flink by default should use random ports. IMHO, allowing components from different hosts to bi-directionally communicate with each other via the loopback address does not sounds like a common use case. It seems to be a violation of the loopback address semantics. This is probably more a problem of the proxy software, rather than Flink. I'd love to help you find a workaround to get things work. However, as one of the maintainers of the Flink project, I'd be negative to introduce such a knob for such an unusual use case. > Flink Task Manager skip loopback interface for resource manager registration > > > Key: FLINK-29572 > URL: https://issues.apache.org/jira/browse/FLINK-29572 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.15.2 > Environment: Flink 1.15.2 > Kubernetes with Istio Proxy >Reporter: Kevin Li >Priority: Major > > Currently Flink Task Manager use different local interface to bind to connect > to Resource Manager. First one is Loopback interface. Normally if Job Manager > is running on remote host/container, using loopback interface to connect will > fail and it will pick up correct IP address. > However, if Task Manager is running with some proxy, loopback interface can > connect to remote host as well. This will result 127.0.0.1 reported to > Resource Manager during registration, even Job Manager/Resource Manager runs > on remote host, and problem will happen. For us, only one Task Manager can > register in this case. > I suggest adding configuration to skip Loopback interface check if we know > Job/Resource Manager is running on remote host/container. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wangyang0918 commented on pull request #21178: [BP-1.15][FLINK-29539][Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
wangyang0918 commented on PR #21178: URL: https://github.com/apache/flink/pull/21178#issuecomment-1294356340 Fix the compile error and force pushed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29786) VarianceThresholdSelector Uses FeaturesCol Incorrectly
[ https://issues.apache.org/jira/browse/FLINK-29786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xin updated FLINK-29786: -- Description: Flink ML has both `FeaturesCol` and `InputCol` to be used to set input. The `FeaturesCol` should only be used with `LabelCol`, while `InputCol` can be used more generally. The `VarianceThresholdSelector` should use `InputCol` instead of `FeaturesCol`. was:Flink ML has both `FeaturesCol` and `InputCol` to be used to set input. The `FeaturesCol` should only be used with `LabelCol`, while `InputCol` can be used more generally. The `VarianceThresholdSelector` should use `InputCol` instead of `FeaturesCol`. > VarianceThresholdSelector Uses FeaturesCol Incorrectly > -- > > Key: FLINK-29786 > URL: https://issues.apache.org/jira/browse/FLINK-29786 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Reporter: Jiang Xin >Priority: Major > Fix For: ml-2.2.0 > > > Flink ML has both `FeaturesCol` and `InputCol` to be used to set input. The > `FeaturesCol` should only be used with `LabelCol`, while `InputCol` can be > used more generally. > The `VarianceThresholdSelector` should use `InputCol` instead of > `FeaturesCol`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29786) VarianceThresholdSelector Uses FeaturesCol Incorrectly
[ https://issues.apache.org/jira/browse/FLINK-29786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xin updated FLINK-29786: -- Summary: VarianceThresholdSelector Uses FeaturesCol Incorrectly (was: VarianceThresholdSelector Uses FeaturesCol as Input Param) > VarianceThresholdSelector Uses FeaturesCol Incorrectly > -- > > Key: FLINK-29786 > URL: https://issues.apache.org/jira/browse/FLINK-29786 > Project: Flink > Issue Type: Bug > Components: Library / Machine Learning >Reporter: Jiang Xin >Priority: Major > Fix For: ml-2.2.0 > > > Flink ML has both `FeaturesCol` and `InputCol` to be used to set input. The > `FeaturesCol` should only be used with `LabelCol`, while `InputCol` can be > used more generally. The `VarianceThresholdSelector` should use `InputCol` > instead of `FeaturesCol`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29749) flink info command support dynamic properties
[ https://issues.apache.org/jira/browse/FLINK-29749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625374#comment-17625374 ] Yang Wang commented on FLINK-29749: --- Thanks [~jackylau] for your contribution. > flink info command support dynamic properties > - > > Key: FLINK-29749 > URL: https://issues.apache.org/jira/browse/FLINK-29749 > Project: Flink > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0, 1.15.3, 1.16.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-29749) flink info command support dynamic properties
[ https://issues.apache.org/jira/browse/FLINK-29749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Wang closed FLINK-29749. - Resolution: Fixed Fixed via: master: f8c6a668cd2b887f33a0cf4608de2d6b95c71f03 release-1.16: 38e90428bf7e603fdd353243f1edeba3553af2a3 release-1.15: 1d29f540a0692540a01b951033a8dc04fdb74d4f > flink info command support dynamic properties > - > > Key: FLINK-29749 > URL: https://issues.apache.org/jira/browse/FLINK-29749 > Project: Flink > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0, 1.15.3, 1.16.1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wangyang0918 merged pull request #21166: [FLINK-29749][client] Make 'flink info' command could support dynamic…
wangyang0918 merged PR #21166: URL: https://github.com/apache/flink/pull/21166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29786) VarianceThresholdSelector Uses FeaturesCol as Input Param
Jiang Xin created FLINK-29786: - Summary: VarianceThresholdSelector Uses FeaturesCol as Input Param Key: FLINK-29786 URL: https://issues.apache.org/jira/browse/FLINK-29786 Project: Flink Issue Type: Bug Components: Library / Machine Learning Reporter: Jiang Xin Fix For: ml-2.2.0 Flink ML has both `FeaturesCol` and `InputCol` to be used to set input. The `FeaturesCol` should only be used with `LabelCol`, while `InputCol` can be used more generally. The `VarianceThresholdSelector` should use `InputCol` instead of `FeaturesCol`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #156: [FLINK-29323] Refine Transformer for VectorAssembler
yunfengzhou-hub commented on code in PR #156:
URL: https://github.com/apache/flink-ml/pull/156#discussion_r1007512230
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/vectorassembler/VectorAssembler.java:
##
@@ -47,10 +47,16 @@
/**
* A Transformer which combines a given list of input columns into a vector
column. Types of input
- * columns must be either vector or numerical value.
+ * columns must be either vector or numerical types. The operator deals with
null values or records
+ * with wrong sizes according to the strategy specified by the {@link
HasHandleInvalid} parameter as
+ * follows:
*
- * The `keep` option of {@link HasHandleInvalid} means that we output bad
rows with output column
- * set to null.
+ * The `keep` option means that if the input column data is NaN, then it
keeps this value and if
+ * data is null vector, then uses a NaN vector to replace it.
+ *
+ * The `skip` option means that it filters out rows with invalid elements.
+ *
+ * The `error` option means that it throws an error exception when meeting
some invalid data.
Review Comment:
The following tags could better format the JavaDoc.
```html
AAA
BBB
CCC
```
##
flink-ml-lib/src/test/java/org/apache/flink/ml/feature/VectorAssemblerTest.java:
##
@@ -45,13 +45,16 @@
Review Comment:
Let's further add test cases to verify different handleInvalid strategies
when the input vectors are valid (no null or NaN values), but the size of the
vectors does not match with that set in `inputSizes`.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/vectorassembler/VectorAssembler.java:
##
@@ -74,64 +81,96 @@ public Table[] transform(Table... inputs) {
DataStream output =
tEnv.toDataStream(inputs[0])
.flatMap(
-new AssemblerFunc(getInputCols(),
getHandleInvalid()),
+new AssemblerFunction(
+getInputCols(), getHandleInvalid(),
getInputSizes()),
outputTypeInfo);
Table outputTable = tEnv.fromDataStream(output);
return new Table[] {outputTable};
}
-private static class AssemblerFunc implements FlatMapFunction {
+private static class AssemblerFunction implements FlatMapFunction {
private final String[] inputCols;
private final String handleInvalid;
+private final Integer[] inputSizes;
+private int vectorSize = 0;
+private final boolean keepInvalid;
-public AssemblerFunc(String[] inputCols, String handleInvalid) {
+public AssemblerFunction(String[] inputCols, String handleInvalid,
Integer[] sizeArray) {
this.inputCols = inputCols;
this.handleInvalid = handleInvalid;
+this.inputSizes = sizeArray;
+for (Integer inputSize : inputSizes) {
+vectorSize += inputSize;
+}
+keepInvalid = handleInvalid.equals(HasHandleInvalid.KEEP_INVALID);
}
@Override
public void flatMap(Row value, Collector out) {
int nnz = 0;
-int vectorSize = 0;
try {
-for (String inputCol : inputCols) {
-Object object = value.getField(inputCol);
-Preconditions.checkNotNull(object, "Input column value
should not be null.");
+for (int i = 0; i < inputCols.length; ++i) {
+Object object = value.getField(inputCols[i]);
+if (object == null) {
+if (keepInvalid) {
+if (inputSizes[i] > 1) {
+DenseVector tmpVec = new
DenseVector(inputSizes[i]);
+for (int j = 0; j < inputSizes[i]; ++j) {
+tmpVec.values[j] = Double.NaN;
+}
+object = tmpVec;
+} else {
+object = Double.NaN;
+}
+value.setField(inputCols[i], object);
+} else {
+throw new RuntimeException(
+"Input column value is null. Please check
the input data or using handleInvalid = 'keep'.");
+}
+}
if (object instanceof Number) {
+if (Double.isNaN(((Number) object).doubleValue()) &&
!keepInvalid) {
+throw new RuntimeException(
+"Encountered NaN while assembling a row
with handleInvalid = 'error'. Consider "
++ "removing NaNs from dataset or
using handleInvalid = 'keep'
[jira] [Created] (FLINK-29785) Upgrade Flink Elasticsearch-7 connector elasticsearch.version to 7.17.0
Elkhan Dadashov created FLINK-29785: --- Summary: Upgrade Flink Elasticsearch-7 connector elasticsearch.version to 7.17.0 Key: FLINK-29785 URL: https://issues.apache.org/jira/browse/FLINK-29785 Project: Flink Issue Type: Improvement Components: Connectors / ElasticSearch Reporter: Elkhan Dadashov `elasticsearch-7` connector uses ElasticSearch `7.10.2` version. [https://github.com/apache/flink-connector-elasticsearch/blob/main/flink-connector-elasticsearch7/pom.xml#L39] When ElasticSearch server side is `8.X.X` then, `elasticsearch-7` connector does not work. For Example for ElasticSearch `8.2.2` version on the server side, the minimum required version on the `elasticsearch-7` Flink connector side is `"minimum_wire_compatibility_version" : "7.17.0"`. As of today `elasticsearch-8` does not exist yet. [ Add Elasticsearch 8.0 support Jira ticket |https://issues.apache.org/jira/browse/FLINK-26088 ] . With this intermediare step - upgrading `elasticsearch-7` connector `elasticsearch.version` to 7.17.0 can help Flink users still ingest into ElasticSearch which has `8.X.X` version deployed on the server side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] ericxiao251 commented on pull request #21077: [FLINK-29498] Add Scala Async Retry Strategies and ResultPredicates Helper Classes
ericxiao251 commented on PR #21077: URL: https://github.com/apache/flink/pull/21077#issuecomment-1294158906 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-29783) Flaky test: KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
[
https://issues.apache.org/jira/browse/FLINK-29783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625337#comment-17625337
]
Martijn Visser edited comment on FLINK-29783 at 10/27/22 9:14 PM:
--
Duplicate of FLINK-24119
was (Author: martijnvisser):
Duplicate of FLINk-24119
> Flaky test:
> KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
> -
>
> Key: FLINK-29783
> URL: https://issues.apache.org/jira/browse/FLINK-29783
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.17.0
>Reporter: Gabor Somogyi
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Oct 27 15:07:54 java.lang.AssertionError: Create test topic :
> partition_failure_recovery_EventTime failed,
> org.apache.kafka.common.errors.TopicExistsException: Topic
> 'partition_failure_recovery_EventTime' already exists.
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:207)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:217)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecovery(KafkaShuffleExactlyOnceITCase.java:158)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime(KafkaShuffleExactlyOnceITCase.java:101)
> Oct 27 15:07:54 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Oct 27 15:07:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Oct 27 15:07:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Oct 27 15:07:54 at java.lang.reflect.Method.invoke(Method.java:498)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Oct 27 15:07:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Oct 27 15:07:54 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Oct 27 15:07:54 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29783) Flaky test: KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
[
https://issues.apache.org/jira/browse/FLINK-29783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-29783.
--
Resolution: Duplicate
Duplicate of FLINk-24119
> Flaky test:
> KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
> -
>
> Key: FLINK-29783
> URL: https://issues.apache.org/jira/browse/FLINK-29783
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.17.0
>Reporter: Gabor Somogyi
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Oct 27 15:07:54 java.lang.AssertionError: Create test topic :
> partition_failure_recovery_EventTime failed,
> org.apache.kafka.common.errors.TopicExistsException: Topic
> 'partition_failure_recovery_EventTime' already exists.
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:207)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:217)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecovery(KafkaShuffleExactlyOnceITCase.java:158)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime(KafkaShuffleExactlyOnceITCase.java:101)
> Oct 27 15:07:54 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Oct 27 15:07:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Oct 27 15:07:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Oct 27 15:07:54 at java.lang.reflect.Method.invoke(Method.java:498)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Oct 27 15:07:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Oct 27 15:07:54 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Oct 27 15:07:54 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29716) Separate slf4j jar in the lib folder from the distribution
[
https://issues.apache.org/jira/browse/FLINK-29716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexis Sarda-Espinosa closed FLINK-29716.
-
Resolution: Won't Do
> Separate slf4j jar in the lib folder from the distribution
> --
>
> Key: FLINK-29716
> URL: https://issues.apache.org/jira/browse/FLINK-29716
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.15.2
>Reporter: Alexis Sarda-Espinosa
>Priority: Major
>
> Flink's binary distribution includes several jars under the {{lib}} folder,
> which has individual jars for all log4j artifacts. This makes it relatively
> easy to swap out those logging jars when necessary, for example when critical
> vulnerabilities are found (as was recently the case).
> With SLF4J 2.+, some breaking changes mean that many implementations are not
> directly backwards compatible, see for example the [notes for
> log4j2|https://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html].
> This means that, in the future, if swapping logging jars were necessary, the
> SLF4J jar might have to be changed as well.
> Right now the SLF4J jar is not included separately in the distribution, I
> believe it's packed inside the {{flink-dist}} jar, although I'm not sure. It
> would be better to separate that as it is done for the default log4j2 jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29716) Separate slf4j jar in the lib folder from the distribution
[
https://issues.apache.org/jira/browse/FLINK-29716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625336#comment-17625336
]
Alexis Sarda-Espinosa commented on FLINK-29716:
---
I see, all right.
> Separate slf4j jar in the lib folder from the distribution
> --
>
> Key: FLINK-29716
> URL: https://issues.apache.org/jira/browse/FLINK-29716
> Project: Flink
> Issue Type: Improvement
>Affects Versions: 1.15.2
>Reporter: Alexis Sarda-Espinosa
>Priority: Major
>
> Flink's binary distribution includes several jars under the {{lib}} folder,
> which has individual jars for all log4j artifacts. This makes it relatively
> easy to swap out those logging jars when necessary, for example when critical
> vulnerabilities are found (as was recently the case).
> With SLF4J 2.+, some breaking changes mean that many implementations are not
> directly backwards compatible, see for example the [notes for
> log4j2|https://logging.apache.org/log4j/2.x/log4j-slf4j-impl/index.html].
> This means that, in the future, if swapping logging jars were necessary, the
> SLF4J jar might have to be changed as well.
> Right now the SLF4J jar is not included separately in the distribution, I
> believe it's packed inside the {{flink-dist}} jar, although I'm not sure. It
> would be better to separate that as it is done for the default log4j2 jars.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] ericxiao251 commented on pull request #21077: [FLINK-29498] Add Scala Async Retry Strategies and ResultPredicates Helper Classes
ericxiao251 commented on PR #21077: URL: https://github.com/apache/flink/pull/21077#issuecomment-1294048147 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29572) Flink Task Manager skip loopback interface for resource manager registration
[ https://issues.apache.org/jira/browse/FLINK-29572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625277#comment-17625277 ] Kevin Li commented on FLINK-29572: -- It will work if we configure different ports for each different task manager. But that will be cumbersome. If you have 10 task manager, you need to create 10 different deployments for each of them. Also autoscale could be issue too. Rather than you have one deployment with 10 replicas and they can scale up and down. I downgrade my Flink to 1.14.6 and it works fine. Looks like it was introduced by FLINK-24474. > Flink Task Manager skip loopback interface for resource manager registration > > > Key: FLINK-29572 > URL: https://issues.apache.org/jira/browse/FLINK-29572 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.15.2 > Environment: Flink 1.15.2 > Kubernetes with Istio Proxy >Reporter: Kevin Li >Priority: Major > > Currently Flink Task Manager use different local interface to bind to connect > to Resource Manager. First one is Loopback interface. Normally if Job Manager > is running on remote host/container, using loopback interface to connect will > fail and it will pick up correct IP address. > However, if Task Manager is running with some proxy, loopback interface can > connect to remote host as well. This will result 127.0.0.1 reported to > Resource Manager during registration, even Job Manager/Resource Manager runs > on remote host, and problem will happen. For us, only one Task Manager can > register in this case. > I suggest adding configuration to skip Loopback interface check if we know > Job/Resource Manager is running on remote host/container. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] gaborgsomogyi commented on pull request #21180: [FLINK-29783][tests] Add some random to KafkaShuffleExactlyOnceITCase topic names
gaborgsomogyi commented on PR #21180: URL: https://github.com/apache/flink/pull/21180#issuecomment-1293940368 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29784) Build fails with There is at least one incompatibility: org.apache.flink.api.connector.source.SourceReader
Sergey Nuyanzin created FLINK-29784:
---
Summary: Build fails with There is at least one incompatibility:
org.apache.flink.api.connector.source.SourceReader
Key: FLINK-29784
URL: https://issues.apache.org/jira/browse/FLINK-29784
Project: Flink
Issue Type: Bug
Components: API / Core
Affects Versions: 1.17.0
Reporter: Sergey Nuyanzin
currently {{./mvnw clean install -DskipTests}} fails with
{noformat}
[ERROR] Failed to execute goal
io.github.zentol.japicmp:japicmp-maven-plugin:0.16.0_m325:cmp (default) on
project flink-core: There is at least one incompatibility:
org.apache.flink.api.connector.source.SourceReader.pauseOrResumeSplits(java.util.Collection,java.util.Collection):METHOD_NEW_DEFAULT
-> [Help 1]
[ERROR]
{noformat}
It starts failing after this commit
https://github.com/apache/flink/commit/82567cc9e9a23a2b6ca41f433c4b9310c0075767
removal of that commit fixes the behavior
// cc [~hxbks2ks] may be you know more details about that
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] gaborgsomogyi commented on pull request #21160: [FLINK-29761][runtime][security] Simplify HadoopModule
gaborgsomogyi commented on PR #21160: URL: https://github.com/apache/flink/pull/21160#issuecomment-1293904151 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21180: [FLINK-29783][tests] Add some random to KafkaShuffleExactlyOnceITCase topic names
gaborgsomogyi commented on PR #21180: URL: https://github.com/apache/flink/pull/21180#issuecomment-1293903766 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21147: [FLINK-28330][runtime][security] Remove old delegation token framework code
gaborgsomogyi commented on PR #21147: URL: https://github.com/apache/flink/pull/21147#issuecomment-1293897838 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21181: [hotfix][doc] `FlinkKafkaConsumer` will be removed with Flink 1.17 instead of 1.15
flinkbot commented on PR #21181: URL: https://github.com/apache/flink/pull/21181#issuecomment-1293882353 ## CI report: * 85079793ed48230d97a1921862c7d28304f34fc7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingGe opened a new pull request, #21181: [hotfix][doc] `FlinkKafkaConsumer` will be removed with Flink 1.17 instead of 1.15
JingGe opened a new pull request, #21181: URL: https://github.com/apache/flink/pull/21181 ## What is the purpose of the change Back port #21172 to master: Change the docs: FlinkKafkaConsumer is deprecated and will be removed with Flink 1.17 instead of 1.15 since Flink 1.16 has code freeze and will be released soon. ## Brief change log - kafka.md - zh kafka.md ## Verifying this change This change is a trivial doc update without any test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sap1ens commented on a diff in pull request #20475: [FLINK-28842][Connector/Kafka] Add client.id.prefix for the KafkaSink
sap1ens commented on code in PR #20475:
URL: https://github.com/apache/flink/pull/20475#discussion_r1007187247
##
flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaWriter.java:
##
@@ -398,6 +407,11 @@ private void registerMetricSync() {
});
}
+private String createProducerClientId(Properties props, int subtaskId) {
+String prefix =
props.getProperty(KafkaSinkOptions.CLIENT_ID_PREFIX.key());
+return prefix + "-" + subtaskId;
Review Comment:
Right, updated!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] SaravananArumugam-Opus commented on a diff in pull request #20922: [DRAFT] Apply changes for calcite 1.3x
SaravananArumugam-Opus commented on code in PR #20922: URL: https://github.com/apache/flink/pull/20922#discussion_r1007141005 ## flink-table/pom.xml: ## @@ -76,10 +76,10 @@ under the License. - 1.26.0 + 1.32.0 3.0.11 Review Comment: Got it… Let me give a try … -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21180: [FLINK-29783][Tests] Add some random to KafkaShuffleExactlyOnceITCasetopic names
flinkbot commented on PR #21180: URL: https://github.com/apache/flink/pull/21180#issuecomment-1293803451 ## CI report: * ccc5e22c6617909e90705c11e4e28dfc12d9310b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaborgsomogyi commented on pull request #21180: [FLINK-29783][Tests] Add some random to KafkaShuffleExactlyOnceITCasetopic names
gaborgsomogyi commented on PR #21180: URL: https://github.com/apache/flink/pull/21180#issuecomment-1293804227 Let's see whether it helps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29783) Flaky test: KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
[
https://issues.apache.org/jira/browse/FLINK-29783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-29783:
---
Labels: pull-request-available (was: )
> Flaky test:
> KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
> -
>
> Key: FLINK-29783
> URL: https://issues.apache.org/jira/browse/FLINK-29783
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.17.0
>Reporter: Gabor Somogyi
>Priority: Major
> Labels: pull-request-available
>
> {code:java}
> Oct 27 15:07:54 java.lang.AssertionError: Create test topic :
> partition_failure_recovery_EventTime failed,
> org.apache.kafka.common.errors.TopicExistsException: Topic
> 'partition_failure_recovery_EventTime' already exists.
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:207)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:217)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecovery(KafkaShuffleExactlyOnceITCase.java:158)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime(KafkaShuffleExactlyOnceITCase.java:101)
> Oct 27 15:07:54 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Oct 27 15:07:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Oct 27 15:07:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Oct 27 15:07:54 at java.lang.reflect.Method.invoke(Method.java:498)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Oct 27 15:07:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Oct 27 15:07:54 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Oct 27 15:07:54 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] gaborgsomogyi opened a new pull request, #21180: [FLINK-29783][Tests] Add some random to KafkaShuffleExactlyOnceITCasetopic names
gaborgsomogyi opened a new pull request, #21180:
URL: https://github.com/apache/flink/pull/21180
## What is the purpose of the change
`KafkaShuffleExactlyOnceITCasetopic.testAssignedToPartitionFailureRecoveryEventTime`
fails with the following exception:
```
{code:java}
Oct 27 15:07:54 java.lang.AssertionError: Create test topic :
partition_failure_recovery_EventTime failed,
org.apache.kafka.common.errors.TopicExistsException: Topic
'partition_failure_recovery_EventTime' already exists.
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:207)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:217)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecovery(KafkaShuffleExactlyOnceITCase.java:158)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime(KafkaShuffleExactlyOnceITCase.java:101)
Oct 27 15:07:54 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Oct 27 15:07:54 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Oct 27 15:07:54 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Oct 27 15:07:54 at java.lang.reflect.Method.invoke(Method.java:498)
Oct 27 15:07:54 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Oct 27 15:07:54 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Oct 27 15:07:54 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
Oct 27 15:07:54 at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Oct 27 15:07:54 at java.lang.Thread.run(Thread.java:748)
{code}
```
I don't see any reason why multiple times the same topic would be created so
what I assume is that somehow the test is executed multiple times within the
JVM and Kafka instance is class bounded.
In the current PR I've made a quick fix to make this disappear and allow
peoples to work on features.
## Brief change log
Added some random to KafkaShuffleExactlyOnceITCasetopic names.
## Verifying this change
Existing unit tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? not applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20922: [DRAFT] Apply changes for calcite 1.3x
snuyanzin commented on code in PR #20922: URL: https://github.com/apache/flink/pull/20922#discussion_r1007127903 ## flink-table/pom.xml: ## @@ -76,10 +76,10 @@ under the License. - 1.26.0 + 1.32.0 3.0.11 Review Comment: yes, i know however from another side current there is a huge amount of tests which should be passed. With current changes they are passing. With janino 3.1.x no. Once you have them passing - feel free to update the version -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29783) Flaky test: KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
Gabor Somogyi created FLINK-29783:
-
Summary: Flaky test:
KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
Key: FLINK-29783
URL: https://issues.apache.org/jira/browse/FLINK-29783
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.17.0
Reporter: Gabor Somogyi
{code:java}
Oct 27 15:07:54 java.lang.AssertionError: Create test topic :
partition_failure_recovery_EventTime failed,
org.apache.kafka.common.errors.TopicExistsException: Topic
'partition_failure_recovery_EventTime' already exists.
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:207)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:217)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecovery(KafkaShuffleExactlyOnceITCase.java:158)
Oct 27 15:07:54 at
org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime(KafkaShuffleExactlyOnceITCase.java:101)
Oct 27 15:07:54 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Oct 27 15:07:54 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Oct 27 15:07:54 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Oct 27 15:07:54 at java.lang.reflect.Method.invoke(Method.java:498)
Oct 27 15:07:54 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
Oct 27 15:07:54 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Oct 27 15:07:54 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
Oct 27 15:07:54 at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
Oct 27 15:07:54 at
java.util.concurrent.FutureTask.run(FutureTask.java:266)
Oct 27 15:07:54 at java.lang.Thread.run(Thread.java:748)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29783) Flaky test: KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
[
https://issues.apache.org/jira/browse/FLINK-29783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625201#comment-17625201
]
Gabor Somogyi commented on FLINK-29783:
---
It's super annoying to restart CI and hope that it works. Trying to add a quick
fix.
> Flaky test:
> KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime
> -
>
> Key: FLINK-29783
> URL: https://issues.apache.org/jira/browse/FLINK-29783
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.17.0
>Reporter: Gabor Somogyi
>Priority: Major
>
> {code:java}
> Oct 27 15:07:54 java.lang.AssertionError: Create test topic :
> partition_failure_recovery_EventTime failed,
> org.apache.kafka.common.errors.TopicExistsException: Topic
> 'partition_failure_recovery_EventTime' already exists.
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.createTestTopic(KafkaTestEnvironmentImpl.java:207)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.createTestTopic(KafkaTestEnvironment.java:97)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase.createTestTopic(KafkaTestBase.java:217)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecovery(KafkaShuffleExactlyOnceITCase.java:158)
> Oct 27 15:07:54 at
> org.apache.flink.streaming.connectors.kafka.shuffle.KafkaShuffleExactlyOnceITCase.testAssignedToPartitionFailureRecoveryEventTime(KafkaShuffleExactlyOnceITCase.java:101)
> Oct 27 15:07:54 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Oct 27 15:07:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Oct 27 15:07:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Oct 27 15:07:54 at java.lang.reflect.Method.invoke(Method.java:498)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Oct 27 15:07:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Oct 27 15:07:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Oct 27 15:07:54 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Oct 27 15:07:54 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Oct 27 15:07:54 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] SaravananArumugam-Opus commented on a diff in pull request #20922: [DRAFT] Apply changes for calcite 1.3x
SaravananArumugam-Opus commented on code in PR #20922: URL: https://github.com/apache/flink/pull/20922#discussion_r1007121946 ## flink-table/pom.xml: ## @@ -76,10 +76,10 @@ under the License. - 1.26.0 + 1.32.0 3.0.11 Review Comment: Calcite - 1.32.0 has dependency on Janino 3.1.8. (\) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21179: [BP-1.16][FLINK-29539][Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
flinkbot commented on PR #21179: URL: https://github.com/apache/flink/pull/21179#issuecomment-1293778963 ## CI report: * 1f30c93213d37ea19210229027af68bcefe51a2f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] snuyanzin commented on a diff in pull request #20922: [DRAFT] Apply changes for calcite 1.3x
snuyanzin commented on code in PR #20922: URL: https://github.com/apache/flink/pull/20922#discussion_r1007111394 ## flink-table/pom.xml: ## @@ -76,10 +76,10 @@ under the License. - 1.26.0 + 1.32.0 3.0.11 Review Comment: Before doing such update it's required to make it working... right now it doesn't work... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SaravananArumugam-Opus commented on a diff in pull request #20922: [DRAFT] Apply changes for calcite 1.3x
SaravananArumugam-Opus commented on code in PR #20922: URL: https://github.com/apache/flink/pull/20922#discussion_r1007109247 ## flink-table/pom.xml: ## @@ -76,10 +76,10 @@ under the License. - 1.26.0 + 1.32.0 3.0.11 Review Comment: Update Janino version to 3.1.8 (https://github.com/apache/calcite/blob/calcite-1.32.0/gradle.properties#L118) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21178: [BP-1.15][FLINK-29539][Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
flinkbot commented on PR #21178: URL: https://github.com/apache/flink/pull/21178#issuecomment-1293768219 ## CI report: * 096aa77e435541c36e0442b2892894e4e6c419f3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29654) Vulnerable libraries - Flink 1.15.2
[ https://issues.apache.org/jira/browse/FLINK-29654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17625193#comment-17625193 ] nagasudhakar commented on FLINK-29654: -- Thanks [~martijnvisser] > Vulnerable libraries - Flink 1.15.2 > --- > > Key: FLINK-29654 > URL: https://issues.apache.org/jira/browse/FLINK-29654 > Project: Flink > Issue Type: Bug > Components: Build System >Affects Versions: 1.15.2 >Reporter: nagasudhakar >Priority: Major > > Hi, our organisation ran a security scan on Flink-1.15.2 release and found > the following vulnerable open source libraries being used - > JDOM1.1 > kryo2.24.0 > libnetty-3.9-java3.9.0.Final > Netty Project3.10.6.Final > Play2.6.11 > Apache Tika1.28.1 > Apache Avro1.7.7 > Apache Kafka2.8.1 > The recommended versions for these libraries are - > JDOM2.0.2 > kryo-5.5.0 > libnetty-3.9-java3.9.9.Final > Netty Project 5.0.0.Final > Play2.8.16 > Apache Tika2.4.1 > Apache Avro1.8.2 > Apache Kafka2.8.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] carloscastrojumo commented on pull request #20982: [FLINK-29539] [Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
carloscastrojumo commented on PR #20982: URL: https://github.com/apache/flink/pull/20982#issuecomment-1293765226 > @carloscastrojumo Could you please create the backport PR for release-1.15 and release-1.16? Done @wangyang0918 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] carloscastrojumo opened a new pull request, #21179: [BP-1.16][FLINK-29539][Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
carloscastrojumo opened a new pull request, #21179: URL: https://github.com/apache/flink/pull/21179 This PR is a backport of PR https://github.com/apache/flink/pull/20982 that relates to [FLINK-29539](https://issues.apache.org/jira/browse/FLINK-29539) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] carloscastrojumo opened a new pull request, #21178: [BP-1.15][FLINK-29539][Deployment/Kubernetes] dnsPolicy in FlinkPod is not overridable
carloscastrojumo opened a new pull request, #21178: URL: https://github.com/apache/flink/pull/21178 This PR is a backport of PR https://github.com/apache/flink/pull/20982 that relates to [FLINK-29539](https://issues.apache.org/jira/browse/FLINK-29539) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
