[jira] [Updated] (FLINK-29424) Add Pull Request Template for flink-ml Project
[ https://issues.apache.org/jira/browse/FLINK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29424: --- Labels: pull-request-available (was: ) > Add Pull Request Template for flink-ml Project > -- > > Key: FLINK-29424 > URL: https://issues.apache.org/jira/browse/FLINK-29424 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Jiang Xin >Priority: Major > Labels: pull-request-available > > Add Pull Request Template for flink-ml Project -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] jiangxin369 opened a new pull request, #159: [FLINK-29424] Add Pull Request Template
jiangxin369 opened a new pull request, #159: URL: https://github.com/apache/flink-ml/pull/159 ## What is the purpose of the change Add Pull Request Template for Flink ML ## Brief change log - Add Pull Request Template for Flink ML ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] jiangxin369 commented on a diff in pull request #158: [FLINK-29409] Add Transformer and Estimator for VarianceThresholdSelector
jiangxin369 commented on code in PR #158: URL: https://github.com/apache/flink-ml/pull/158#discussion_r980765655 ## flink-ml-lib/src/main/java/org/apache/flink/ml/feature/variancethresholdselector/VarianceThresholdSelectorParams.java: ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.feature.variancethresholdselector; + +import org.apache.flink.ml.param.DoubleParam; +import org.apache.flink.ml.param.Param; +import org.apache.flink.ml.param.ParamValidators; + +/** + * * Params of VarianceThresholdSelectorModel. Review Comment: Thanks for your code review, I'll remove it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #158: [FLINK-29409] Add Transformer and Estimator for VarianceThresholdSelector
yunfengzhou-hub commented on code in PR #158: URL: https://github.com/apache/flink-ml/pull/158#discussion_r980705519 ## flink-ml-lib/src/main/java/org/apache/flink/ml/feature/variancethresholdselector/VarianceThresholdSelectorParams.java: ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.feature.variancethresholdselector; + +import org.apache.flink.ml.param.DoubleParam; +import org.apache.flink.ml.param.Param; +import org.apache.flink.ml.param.ParamValidators; + +/** + * * Params of VarianceThresholdSelectorModel. Review Comment: nit: "Params of VarianceThresholdSelector". The `*` before "Params" should be removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #158: [FLINK-29409] Add Transformer and Estimator for VarianceThresholdSelector
yunfengzhou-hub commented on code in PR #158: URL: https://github.com/apache/flink-ml/pull/158#discussion_r980705519 ## flink-ml-lib/src/main/java/org/apache/flink/ml/feature/variancethresholdselector/VarianceThresholdSelectorParams.java: ## @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.feature.variancethresholdselector; + +import org.apache.flink.ml.param.DoubleParam; +import org.apache.flink.ml.param.Param; +import org.apache.flink.ml.param.ParamValidators; + +/** + * * Params of VarianceThresholdSelectorModel. Review Comment: nit: "Params of VarianceThresholdSelector". The `*` before "Params` should be removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-28256) Move the write and prepareCommit logic of AbstractTableWrite to FileStoreWrite
[ https://issues.apache.org/jira/browse/FLINK-28256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-28256: Assignee: Shammon > Move the write and prepareCommit logic of AbstractTableWrite to FileStoreWrite > -- > > Key: FLINK-28256 > URL: https://issues.apache.org/jira/browse/FLINK-28256 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Shammon >Priority: Minor > Labels: pull-request-available > Fix For: table-store-0.3.0 > > > TableWrite can be a simple key value converter wrapper build on > FileStoreWrite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-28256) Move the write and prepareCommit logic of AbstractTableWrite to FileStoreWrite
[ https://issues.apache.org/jira/browse/FLINK-28256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-28256: --- Labels: pull-request-available (was: ) > Move the write and prepareCommit logic of AbstractTableWrite to FileStoreWrite > -- > > Key: FLINK-28256 > URL: https://issues.apache.org/jira/browse/FLINK-28256 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Minor > Labels: pull-request-available > Fix For: table-store-0.3.0 > > > TableWrite can be a simple key value converter wrapper build on > FileStoreWrite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] zjureel opened a new pull request, #305: [FLINK-28256] Move the write and prepareCommit logic of AbstractTableWrite to FileStoreWrite
zjureel opened a new pull request, #305: URL: https://github.com/apache/flink-table-store/pull/305 Currently `TableWrite` creates and manages the `RecordWriter`, which is created by `FileStoreWrite`. As `TableWrite` corresponds to `FileStoreTable` and `FileStoreWrite` corresponds to `FileStore`, it's better to move the `write` and `prepareCommit` logic of `AbstractTableWrite` to `FileStoreWrite` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29426) Flink ML 2.1.0 Installation
Tony Sun created FLINK-29426:
Summary: Flink ML 2.1.0 Installation
Key: FLINK-29426
URL: https://issues.apache.org/jira/browse/FLINK-29426
Project: Flink
Issue Type: Bug
Components: Quickstarts
Affects Versions: ml-2.1.0
Environment: Java version: 11.0.16
Flink version: 1.15.2
Reporter: Tony Sun
Trying to go through Flink-ML quickstart but having an issue with building
Flink ML's source code.
Steps to reproduce:
1. sudo wget
https://dlcdn.apache.org/flink/flink-ml-2.1.0/flink-ml-2.1.0-src.tgz
2. sudo tar -xzf flink-ml-2.1.0-src.tgz
3. mvn clean package -DskipTests -Dcheckstyle.skip
{quote}Failed to execute goal
org.apache.maven.plugins:maven-checkstyle-plugin:2.17:check (validate) on
project flink-ml-parent: Unable to create output stream:
/usr/src/flink-ml-2.1.0/target/checkstyle-result.xml:
/usr/src/flink-ml-2.1.0/target/checkstyle-result.xml (No such file or
directory) -> [Help 1]
{quote}
It doesn't seem like `target` is a folder in `flink-ml-2.1.0`.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-docker] HuangXingBo closed pull request #136: Update Dockerfiles for 1.14.6 release
HuangXingBo closed pull request #136: Update Dockerfiles for 1.14.6 release URL: https://github.com/apache/flink-docker/pull/136 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-docker] HuangXingBo closed pull request #135: Add GPG key for 1.14.6 release
HuangXingBo closed pull request #135: Add GPG key for 1.14.6 release URL: https://github.com/apache/flink-docker/pull/135 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29425) Hybrid full spilling strategy triggering spilling frequently
Weijie Guo created FLINK-29425: -- Summary: Hybrid full spilling strategy triggering spilling frequently Key: FLINK-29425 URL: https://issues.apache.org/jira/browse/FLINK-29425 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.16.0 Reporter: Weijie Guo In hybrid shuffle mode, we have an internal config option 'DEFAULT_FULL_STRATEGY_NUM_BUFFERS_TRIGGER_SPILLED' to control spilling frequency in the full spilling strategy. Unfortunately, the default value(10) is too small. As a result, frequent disk spilling calls are observed in the TPC-DS test, which seriously affects performance. When we increase the value, the query performance is improved significantly. We should set a more reasonable default value, or adopt an adaptive strategy to determine the spilling frequency. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-19822) Remove redundant shuffle for streaming
[
https://issues.apache.org/jira/browse/FLINK-19822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609812#comment-17609812
]
Alexander Smirnov commented on FLINK-19822:
---
Hi [~godfreyhe]! I'm wondering why this improvement is still open, although it
looks quite useful. Are there any pitfalls in implementation from linked
Github PR?
> Remove redundant shuffle for streaming
> --
>
> Key: FLINK-19822
> URL: https://issues.apache.org/jira/browse/FLINK-19822
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: godfrey he
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> auto-unassigned, pull-request-available
>
> This is similar
> [FLINK-12575|https://issues.apache.org/jira/browse/FLINK-12575], we could
> implement {{satisfyTraits}} method for stream nodes to remove redundant
> shuffle. This could add more possibilities that more operators can be merged
> into multiple input operator.
> Different batch, stream operators require the shuffle keys and the state keys
> must be exactly the same, otherwise the state may be not correct.
> We only support a few operators in this issue, such as Join and regular
> Aggregate. Other operators will be supported in the future.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on a diff in pull request #157: [FLINK-28906] Support windowing in AgglomerativeClustering
lindong28 commented on code in PR #157:
URL: https://github.com/apache/flink-ml/pull/157#discussion_r980594756
##
flink-ml-python/pyflink/ml/lib/clustering/agglomerativeclustering.py:
##
@@ -112,15 +113,20 @@ def compute_full_tree(self):
class AgglomerativeClustering(JavaClusteringAlgoOperator,
_AgglomerativeClusteringParams):
"""
An AlgoOperator that performs a hierarchical clustering using a bottom-up
approach. Each
-observation starts in its own cluster and the clusters are merged together
one by one.
-Users can choose different strategies to merge two clusters by setting
-{@link AgglomerativeClusteringParams#LINKAGE} and different distance
measures by setting
-{@link AgglomerativeClusteringParams#DISTANCE_MEASURE}.
+observation starts in its own cluster and the clusters are merged together
one by one. Users can
+choose different strategies to merge two clusters by setting
+AgglomerativeClusteringParams#LINKAGE and different distance measures by
setting
+AgglomerativeClusteringParams#DISTANCE_MEASURE.
The output contains two tables. The first one assigns one cluster Id
for each data point.
The second one contains the information of merging two clusters at each
step. The data format
of the merging information is (clusterId1, clusterId2, distance,
sizeOfMergedCluster).
+This operator supports the HasWindows parameter, which creates
mini-batches(windows) from
Review Comment:
Update the doc to be consistent with the corresponding Javadoc.
##
flink-ml-python/pyflink/ml/lib/clustering/tests/test_agglomerativeclustering.py:
##
@@ -58,6 +63,18 @@ def setUp(self):
{Vectors.dense(4, 1.5), Vectors.dense(4, 0)}
]
+self.eucliean_ward_count_five_window_as_two_result = [
Review Comment:
eucliean -> euclidean
Same for other lines in this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29424) Add Pull Request Template for flink-ml Project
[ https://issues.apache.org/jira/browse/FLINK-29424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xin updated FLINK-29424: -- Component/s: Library / Machine Learning > Add Pull Request Template for flink-ml Project > -- > > Key: FLINK-29424 > URL: https://issues.apache.org/jira/browse/FLINK-29424 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Jiang Xin >Priority: Major > > Add Pull Request Template for flink-ml Project -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29424) Add Pull Request Template for flink-ml Project
Jiang Xin created FLINK-29424: - Summary: Add Pull Request Template for flink-ml Project Key: FLINK-29424 URL: https://issues.apache.org/jira/browse/FLINK-29424 Project: Flink Issue Type: New Feature Reporter: Jiang Xin Add Pull Request Template for flink-ml Project -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wangyang0918 commented on a diff in pull request #20867: [FLINK-29372] Add suffix to all options that conflict with YAML
wangyang0918 commented on code in PR #20867:
URL: https://github.com/apache/flink/pull/20867#discussion_r980692851
##
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java:
##
@@ -447,7 +450,7 @@ public class KubernetesConfigOptions {
+ Constants.MAIN_CONTAINER_NAME
+ "'. If not explicitly configured, config
option '"
+ KUBERNETES_POD_TEMPLATE_FILE_KEY
-+ "' will be used.");
++ ".default' will be used.");
public static final ConfigOption TASK_MANAGER_POD_TEMPLATE =
Review Comment:
We also need to update the fallback keys.
```
.withFallbackKeys(KUBERNETES_POD_TEMPLATE_FILE_KEY + ".default")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29181) log.system can be congiured by dynamic options
[ https://issues.apache.org/jira/browse/FLINK-29181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-29181: - Issue Type: Improvement (was: New Feature) > log.system can be congiured by dynamic options > -- > > Key: FLINK-29181 > URL: https://issues.apache.org/jira/browse/FLINK-29181 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.3.0, table-store-0.2.1 > > > Now log.system default value is null, it can not be configured by dynamic > option. > We can let default value is 'none'. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29154) Support LookupTableSource for table store
[ https://issues.apache.org/jira/browse/FLINK-29154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-29154: - Issue Type: Improvement (was: New Feature) > Support LookupTableSource for table store > - > > Key: FLINK-29154 > URL: https://issues.apache.org/jira/browse/FLINK-29154 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.3.0, table-store-0.2.1 > > > At present, the bottom layer of Table Store is LSM structure, and it has full > and incremental reading, which can have certain lookup capability. We can > unlock Table Store to implement LookupTableSource. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] jiangxin369 commented on a diff in pull request #158: [FLINK-29409] Add Transformer and Estimator for VarianceThresholdSelector
jiangxin369 commented on code in PR #158:
URL: https://github.com/apache/flink-ml/pull/158#discussion_r980677136
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/variancethresholdselector/VarianceThresholdSelectorModelData.java:
##
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.variancethresholdselector;
+
+import org.apache.flink.api.common.serialization.Encoder;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.connector.file.src.reader.SimpleStreamFormat;
+import org.apache.flink.core.fs.FSDataInputStream;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+
+import com.esotericsoftware.kryo.io.Input;
+import com.esotericsoftware.kryo.io.Output;
+
+import java.io.IOException;
+import java.io.OutputStream;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Model data of {@link VarianceThresholdSelectorModel}.
+ *
+ * This class also provides methods to convert model data from Table to a
data stream, and
+ * classes to save/load model data.
+ */
+public class VarianceThresholdSelectorModelData {
+
+public int numOfFeatures;
+public int[] indices;
Review Comment:
I think there should only be indices in the model data (so do Spark ML and
sklearn), because the model needs to be transformed on different datasets and
it has better performance.
Indeed, it's better to divide `VarianceThresholdSelectorParams` and
`VarianceThresholdSelectorModelParams`, I'll do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29154) Support LookupTableSource for table store
[ https://issues.apache.org/jira/browse/FLINK-29154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-29154: - Fix Version/s: table-store-0.2.1 > Support LookupTableSource for table store > - > > Key: FLINK-29154 > URL: https://issues.apache.org/jira/browse/FLINK-29154 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.3.0, table-store-0.2.1 > > > At present, the bottom layer of Table Store is LSM structure, and it has full > and incremental reading, which can have certain lookup capability. We can > unlock Table Store to implement LookupTableSource. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29154) Support LookupTableSource for table store
[ https://issues.apache.org/jira/browse/FLINK-29154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17600665#comment-17600665 ] Jingsong Lee edited comment on FLINK-29154 at 9/27/22 2:09 AM: --- master: e55dde87296b51132d37fb4265bd3c863f05db40 release-0.2: c24016d512b5c01c04b6976b19ebdbf9a2ba78d4 was (Author: lzljs3620320): master: e55dde87296b51132d37fb4265bd3c863f05db40 > Support LookupTableSource for table store > - > > Key: FLINK-29154 > URL: https://issues.apache.org/jira/browse/FLINK-29154 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.3.0, table-store-0.2.1 > > > At present, the bottom layer of Table Store is LSM structure, and it has full > and incremental reading, which can have certain lookup capability. We can > unlock Table Store to implement LookupTableSource. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29421) Support python 3.10
[
https://issues.apache.org/jira/browse/FLINK-29421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609778#comment-17609778
]
Huang Xingbo commented on FLINK-29421:
--
Currently, pyflink hasn't supported Python3.10. In release-1.16, pyflink will
support 3.9. For 3.10 support, it will probably be in release 1.17.
> Support python 3.10
> ---
>
> Key: FLINK-29421
> URL: https://issues.apache.org/jira/browse/FLINK-29421
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Reporter: Eric Sirianni
>Priority: Minor
>
> The {{apache-flink}} package fails to install on Python 3.10 due to inability
> to compile {{numpy}}
> {noformat}
> numpy/core/src/multiarray/scalartypes.c.src:3242:12: error: too
> few arguments to function ‘_Py_HashDouble’
> 3242 | return
> _Py_HashDouble(npy_half_to_double(((PyHalfScalarObject *)obj)->obval));
> |^~
> In file included from
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/Python.h:77,
> from
> numpy/core/src/multiarray/scalartypes.c.src:3:
>
> /home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/pyhash.h:10:23:
> note: declared here
>10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
> {noformat}
> Numpy issue https://github.com/numpy/numpy/issues/19033
> [Mailing list
> thread|https://lists.apache.org/thread/f4r9hjt1l33xf5ngnswszhnls4cxkk52]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-ml] jiangxin369 commented on a diff in pull request #158: [FLINK-29409] Add Transformer and Estimator for VarianceThresholdSelector
jiangxin369 commented on code in PR #158:
URL: https://github.com/apache/flink-ml/pull/158#discussion_r980656357
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/variancethresholdselector/VarianceThresholdSelectorModel.java:
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.variancethresholdselector;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A Model which removes low-variance data using the model data computed by
{@link
+ * VarianceThresholdSelector}.
+ */
+public class VarianceThresholdSelectorModel
+implements Model,
+
VarianceThresholdSelectorParams {
+
+private final Map, Object> paramMap = new HashMap<>();
+private Table modelDataTable;
+
+public VarianceThresholdSelectorModel() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public VarianceThresholdSelectorModel setModelData(Table... inputs) {
+Preconditions.checkArgument(inputs.length == 1);
+modelDataTable = inputs[0];
+return this;
+}
+
+@Override
+public Table[] getModelData() {
+return new Table[] {modelDataTable};
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void save(String path) throws IOException {
+ReadWriteUtils.saveMetadata(this, path);
+ReadWriteUtils.saveModelData(
+
VarianceThresholdSelectorModelData.getModelDataStream(modelDataTable),
+path,
+new VarianceThresholdSelectorModelData.ModelDataEncoder());
+}
+
+public static VarianceThresholdSelectorModel load(StreamTableEnvironment
tEnv, String path)
+throws IOException {
+VarianceThresholdSelectorModel model =
ReadWriteUtils.loadStageParam(path);
+Table modelDataTable =
+ReadWriteUtils.loadModelData(
+tEnv, path, new
VarianceThresholdSelectorModelData.ModelDataDecoder());
+return model.setModelData(modelDataTable);
+}
+
+@Override
+public Table[] transform(Table... inputs) {
+Preconditions.checkArgument(inputs.length == 1);
+
+StreamTableEnvironment tEnv =
+(StreamTableEnvironment) ((TableImpl)
inputs[0]).getTableEnvironment();
+DataStream data = tEnv.toDataStream(inputs[0]);
+DataStream
varianceThresholdSelectorModel =
+
VarianceThresholdSelectorModelData.getModelDataStream(modelDataTable);
+
+final String broadcastModelKey = "broadcastModelKey";
+RowTypeInfo inputTypeInfo =
TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+RowTypeInfo outputTypeInfo =
+new RowTypeInfo(
+ArrayUtils.addAll(
+inputTypeInfo.getFieldTypes(),
+TypeInformation.of(DenseVector.class)),
+ArrayUtils.addAll(inputTypeInfo.getFieldNames(),
getOutputCol()));
+
+DataStream output =
+BroadcastUtils.withBroadcastStream(
+
[jira] [Closed] (FLINK-29219) CREATE TABLE AS statement blocks SQL client's execution
[ https://issues.apache.org/jira/browse/FLINK-29219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu closed FLINK-29219. --- Fix Version/s: 1.17.0 Resolution: Fixed Fixed in - master: 44009efc5400c6706563fa53335eccf445cf5d0c - release-1.16: 4c126223cbc2dec3f08b50c1b398bdf522747ba9 > CREATE TABLE AS statement blocks SQL client's execution > --- > > Key: FLINK-29219 > URL: https://issues.apache.org/jira/browse/FLINK-29219 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Client >Affects Versions: 1.16.0 >Reporter: Qingsheng Ren >Assignee: dalongliu >Priority: Blocker > Labels: pull-request-available > Fix For: 1.16.0, 1.17.0 > > > When executing CREATE TABLE AS statement to create a sink table in SQL > client, the client could create the table in catalog and submit the job to > cluster successfully, but stops emitting new prompts and accepts new inputs, > and user has to use SIGTERM (Control + C) to forcefully stop the SQL client. > As contrast the behavior of INSERT INTO statement in SQL client is printing > "Job is submitted with JobID " and being ready to accept user's input. > From the log it looks like the client was waiting for the job to finish. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wuchong merged pull request #20880: [FLINK-29219][table] Fix CREATE TABLE AS statement blocks SQL client's execution
wuchong merged PR #20880: URL: https://github.com/apache/flink/pull/20880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong merged pull request #20869: [FLINK-29219][table] Fix CREATE TABLE AS statement blocks SQL client's execution
wuchong merged PR #20869: URL: https://github.com/apache/flink/pull/20869 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #158: [FLINK-29409] Add Transformer and Estimator for VarianceThresholdSelector
yunfengzhou-hub commented on code in PR #158:
URL: https://github.com/apache/flink-ml/pull/158#discussion_r980634965
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/variancethresholdselector/VarianceThresholdSelectorParams.java:
##
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.variancethresholdselector;
+
+import org.apache.flink.ml.common.param.HasFeaturesCol;
+import org.apache.flink.ml.common.param.HasOutputCol;
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+
+/**
+ * * Params of VarianceThresholdSelectorModel.
+ *
+ * @param The class type of this instance.
+ */
+public interface VarianceThresholdSelectorParams extends HasFeaturesCol,
HasOutputCol {
+
+Param VARIANCE_THRESHOLD =
+new DoubleParam(
+"varianceThreshold",
+"Param for variance threshold. Features with "
++ "a variance not greater than this threshold will
be removed.",
+0.0);
Review Comment:
nit: we can add a `ParamValidators.gtEq(0)` here.
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/variancethresholdselector/VarianceThresholdSelectorModel.java:
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.variancethresholdselector;
+
+import org.apache.flink.api.common.functions.RichMapFunction;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.ml.api.Model;
+import org.apache.flink.ml.common.broadcast.BroadcastUtils;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.Vector;
+import org.apache.flink.ml.linalg.Vectors;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A Model which removes low-variance data using the model data computed by
{@link
+ * VarianceThresholdSelector}.
+ */
+public class VarianceThresholdSelectorModel
+implements Model,
+
VarianceThresholdSelectorParams {
+
+private final Map, Object> paramMap = new HashMap<>();
+private Table modelDataTable;
+
+public VarianceThresholdSelectorModel() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public VarianceThresholdSelectorModel setModelData(Table... inputs) {
+Preconditions.checkArgument(inputs.length == 1);
+modelDataTable = inputs[0];
+return this;
+}
+
+@Override
+public Table[] getModelData() {
+return new Table[] {modelDataTable};
+}
+
+@Override
+public Map, Object> getParamMap() {
+return paramMap;
+}
+
+@Override
+public void
[GitHub] [flink-ml] lindong28 commented on pull request #157: [FLINK-28906] Support windowing in AgglomerativeClustering
lindong28 commented on PR #157: URL: https://github.com/apache/flink-ml/pull/157#issuecomment-1258769768 Thanks for the PR! LGTM overall. Can you fix the test failures? @zhipeng93 do you want to take a look at this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29418) Update flink-shaded
[ https://issues.apache.org/jira/browse/FLINK-29418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29418: --- Labels: pull-request-available (was: ) > Update flink-shaded > --- > > Key: FLINK-29418 > URL: https://issues.apache.org/jira/browse/FLINK-29418 > Project: Flink > Issue Type: Technical Debt > Components: BuildSystem / Shaded >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: shaded-1.16 > > > Do a round of updates in flink-shaded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-shaded] zentol opened a new pull request, #112: [FLINK-29418] Everybody gets an upgrade!
zentol opened a new pull request, #112: URL: https://github.com/apache/flink-shaded/pull/112 Except guava. The upgrades were all tested against 1.17-SNAPSHOT. For 1.16 I'm only interested in the Jackson upgrade, which has some slight behavior changes w.r.t. how the result of the docs generation, but we can handle that. Also adds Zookeeper 3.7 / 3.8 modules. I haven't tested those; they either work for 1.17 or we'll have to do another release anyway. IOW, there's no harm in throwing them in now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20903: [FLINK-29402][state backends] Add RocksDB options for DirectIO reads …
flinkbot commented on PR #20903: URL: https://github.com/apache/flink/pull/20903#issuecomment-1258630220 ## CI report: * 70ab91e25623f474698e153622e52d459eaf0530 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29402) Add USE_DIRECT_READ configuration parameter for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-29402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Donatien updated FLINK-29402: - Fix Version/s: 1.16.0 (was: 1.15.2) Affects Version/s: 1.16.0 (was: 1.15.2) Labels: Enhancement pull-request-available pull_request_available rocksdb (was: Enhancement pull-request-available rocksdb) > Add USE_DIRECT_READ configuration parameter for RocksDB > --- > > Key: FLINK-29402 > URL: https://issues.apache.org/jira/browse/FLINK-29402 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Donatien >Priority: Not a Priority > Labels: Enhancement, pull-request-available, > pull_request_available, rocksdb > Fix For: 1.16.0 > > Attachments: directIO-performance-comparison.png > > Original Estimate: 1h > Remaining Estimate: 1h > > RocksDB allows the use of DirectIO for read operations to bypass the Linux > Page Cache. To understand the impact of Linux Page Cache on performance, one > can run a heavy workload on a single-tasked Task Manager with a container > memory limit identical to the TM process memory. Running this same workload > on a TM with no container memory limit will result in better performances but > with the host memory exceeding the TM requirement. > Linux Page Cache are of course useful but can give false results when > benchmarking the Managed Memory used by RocksDB. DirectIO is typically > enabled for benchmarks on working set estimation [Zwaenepoel et > al.|[https://arxiv.org/abs/1702.04323].] > I propose to add a configuration key allowing users to enable the use of > DirectIO for reads thanks to the RocksDB API. This configuration would be > disabled by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29402) Add USE_DIRECT_READ configuration parameter for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-29402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Donatien updated FLINK-29402: - Labels: Enhancement pull-request-available rocksdb (was: Enhancement pull-request-available pull_request_available rocksdb) > Add USE_DIRECT_READ configuration parameter for RocksDB > --- > > Key: FLINK-29402 > URL: https://issues.apache.org/jira/browse/FLINK-29402 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Donatien >Priority: Not a Priority > Labels: Enhancement, pull-request-available, rocksdb > Fix For: 1.16.0 > > Attachments: directIO-performance-comparison.png > > Original Estimate: 1h > Remaining Estimate: 1h > > RocksDB allows the use of DirectIO for read operations to bypass the Linux > Page Cache. To understand the impact of Linux Page Cache on performance, one > can run a heavy workload on a single-tasked Task Manager with a container > memory limit identical to the TM process memory. Running this same workload > on a TM with no container memory limit will result in better performances but > with the host memory exceeding the TM requirement. > Linux Page Cache are of course useful but can give false results when > benchmarking the Managed Memory used by RocksDB. DirectIO is typically > enabled for benchmarks on working set estimation [Zwaenepoel et > al.|[https://arxiv.org/abs/1702.04323].] > I propose to add a configuration key allowing users to enable the use of > DirectIO for reads thanks to the RocksDB API. This configuration would be > disabled by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29402) Add USE_DIRECT_READ configuration parameter for RocksDB
[ https://issues.apache.org/jira/browse/FLINK-29402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29402: --- Labels: Enhancement pull-request-available rocksdb (was: Enhancement rocksdb) > Add USE_DIRECT_READ configuration parameter for RocksDB > --- > > Key: FLINK-29402 > URL: https://issues.apache.org/jira/browse/FLINK-29402 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.15.2 >Reporter: Donatien >Priority: Not a Priority > Labels: Enhancement, pull-request-available, rocksdb > Fix For: 1.15.2 > > Attachments: directIO-performance-comparison.png > > Original Estimate: 1h > Remaining Estimate: 1h > > RocksDB allows the use of DirectIO for read operations to bypass the Linux > Page Cache. To understand the impact of Linux Page Cache on performance, one > can run a heavy workload on a single-tasked Task Manager with a container > memory limit identical to the TM process memory. Running this same workload > on a TM with no container memory limit will result in better performances but > with the host memory exceeding the TM requirement. > Linux Page Cache are of course useful but can give false results when > benchmarking the Managed Memory used by RocksDB. DirectIO is typically > enabled for benchmarks on working set estimation [Zwaenepoel et > al.|[https://arxiv.org/abs/1702.04323].] > I propose to add a configuration key allowing users to enable the use of > DirectIO for reads thanks to the RocksDB API. This configuration would be > disabled by default. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] Donaschmi opened a new pull request, #20903: [FLINK-29402][state backends] Add RocksDB options for DirectIO reads …
Donaschmi opened a new pull request, #20903: URL: https://github.com/apache/flink/pull/20903 ## What is the purpose of the change This pull request adds two configurable RocksDB options enabling the use of DirectIO for reads and background flush and compactions. ## Brief change log - Add `state.backend.rocksdb.use-direct-reads` to enable DirectIO for read operations - Add `state.backend.rocksdb.use-direct-writes` to enable DirectIO for background flush and compactions ## Verifying this change This change is already covered by existing tests, such as *testConfigurableOptionsFromConfig*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
stevenzwu commented on code in PR #20852: URL: https://github.com/apache/flink/pull/20852#discussion_r980390428 ## flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java: ## @@ -863,11 +871,6 @@ public CompletableFuture triggerSavepoint( return schedulerNG.triggerSavepoint(targetDirectory, cancelJob, formatType); } -@Override Review Comment: got it. you added a default impl method in the parent interface. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
stevenzwu commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r980387378
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java:
##
@@ -878,7 +879,8 @@ public CompletableFuture triggerSavepoint(
}
@Override
-public CompletableFuture triggerCheckpoint() {
+public CompletableFuture triggerCheckpoint(
Review Comment:
oh. I didn't realize it is needed in the trigger checkpoint code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] cfedersp commented on pull request #19861: [hotfix] [docs] Fix use cases for HashMapStateBackend
cfedersp commented on PR #19861:
URL: https://github.com/apache/flink/pull/19861#issuecomment-1258459137
Theres also a factual error on this page where it states:
The EmbeddedRocksDBStateBackend holds in-flight data in a RocksDB database
that is (per default) stored in the TaskManager local data directories.
With minimal RocksDB config:
`env.setStateBackend(new EmbeddedRocksDBStateBackend());
env.getCheckpointConfig().setCheckpointStorage("file:///var/data");`
I find the data on the JobManager.
https://user-images.githubusercontent.com/284126/192354886-091dc2ed-6caa-49f1-84c5-675f56767d97.png;>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] afedulov commented on pull request #20865: [FLINK-14896][connectors/kinesis] Shade and relocate Jackson dependen…
afedulov commented on PR #20865: URL: https://github.com/apache/flink/pull/20865#issuecomment-1258433441 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-29423) JobDetails is incorrect OpenAPI spec
[ https://issues.apache.org/jira/browse/FLINK-29423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler reassigned FLINK-29423: Assignee: Chesnay Schepler > JobDetails is incorrect OpenAPI spec > - > > Key: FLINK-29423 > URL: https://issues.apache.org/jira/browse/FLINK-29423 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / REST >Affects Versions: 1.16.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > > The JobDetails use custom serialization, but the introspection ignores that > and analyzes the class as-is, resulting in various fields being documented > that shouldn't be. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29423) JobDetails is incorrect OpenAPI spec
Chesnay Schepler created FLINK-29423: Summary: JobDetails is incorrect OpenAPI spec Key: FLINK-29423 URL: https://issues.apache.org/jira/browse/FLINK-29423 Project: Flink Issue Type: Bug Components: Documentation, Runtime / REST Affects Versions: 1.16.0 Reporter: Chesnay Schepler The JobDetails use custom serialization, but the introspection ignores that and analyzes the class as-is, resulting in various fields being documented that shouldn't be. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r980354599
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java:
##
@@ -853,6 +855,12 @@ public CompletableFuture
requestJob(Time timeout) {
return CompletableFuture.completedFuture(schedulerNG.requestJob());
}
+@Override
+public CompletableFuture triggerCheckpoint(
Review Comment:
There will be no other `triggerCheckpoint` method in this class, that method
is implemented in its parent interface so it will not be defined explicitly
here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r980350376
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java:
##
@@ -863,11 +871,6 @@ public CompletableFuture triggerSavepoint(
return schedulerNG.triggerSavepoint(targetDirectory, cancelJob,
formatType);
}
-@Override
Review Comment:
Double checked the code - the backward compatibility should already been
kept. At this class's parent interface `JobMasterGateway`, this method is
supported as
```
/**
* Triggers taking a checkpoint of the executed job.
*
* @param timeout for the rpc call
* @return Future which is completed with the checkpoint path once
completed
*/
default CompletableFuture triggerCheckpoint(@RpcTimeout final
Time timeout) {
return triggerCheckpoint(null,
timeout).thenApply(CompletedCheckpoint::getExternalPointer);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] leletan commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
leletan commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r980350376
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java:
##
@@ -863,11 +871,6 @@ public CompletableFuture triggerSavepoint(
return schedulerNG.triggerSavepoint(targetDirectory, cancelJob,
formatType);
}
-@Override
Review Comment:
The backward compatibility should already been kept. At this class's parent
interface `JobMasterGateway`, this method is supported as
```
/**
* Triggers taking a checkpoint of the executed job.
*
* @param timeout for the rpc call
* @return Future which is completed with the checkpoint path once
completed
*/
default CompletableFuture triggerCheckpoint(@RpcTimeout final
Time timeout) {
return triggerCheckpoint(null,
timeout).thenApply(CompletedCheckpoint::getExternalPointer);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] stevenzwu commented on a diff in pull request #20852: [FLINK-27101][checkpointing][rest] Add restful API to trigger checkpoints
stevenzwu commented on code in PR #20852:
URL: https://github.com/apache/flink/pull/20852#discussion_r980320317
##
flink-core/src/main/java/org/apache/flink/core/execution/CheckpointBackupType.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.core.execution;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.DescribedEnum;
+import org.apache.flink.configuration.description.InlineElement;
+
+import static org.apache.flink.configuration.description.TextElement.text;
+
+/** Describes the backup type in which a checkpoint should be taken. */
+@PublicEvolving
+public enum CheckpointBackupType implements DescribedEnum {
Review Comment:
agree that `SnapshotTypeImpl` is not a good choice. `Internal` would be ok
for me.
Just to brainstorm more options. what about `CheckpointSnapshotType` and
`SavepointSnapshotType`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] reswqa commented on pull request #20902: [FLINK-29419] fix the problem that testHybridFullExchangesRestart hang.
reswqa commented on PR #20902: URL: https://github.com/apache/flink/pull/20902#issuecomment-1258354359 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29422) Production tests return/argument types do not take transitivity into account
Chesnay Schepler created FLINK-29422:
Summary: Production tests return/argument types do not take
transitivity into account
Key: FLINK-29422
URL: https://issues.apache.org/jira/browse/FLINK-29422
Project: Flink
Issue Type: Technical Debt
Components: API / DataStream
Affects Versions: 1.16.0
Reporter: Chesnay Schepler
In FLINK-29403 I'm marking {{SimpleCondition}} as {{PublicEvolving}}, but the
production tests reject it:
{code:java}
Architecture Violation [Priority: MEDIUM] - Rule 'Return and argument types of
methods annotated with @PublicEvolving must be annotated with
@Public(Evolving).' was violated (1 times):
Sep 26 15:20:12
org.apache.flink.cep.pattern.conditions.SimpleCondition.filter(java.lang.Object,
org.apache.flink.cep.pattern.conditions.IterativeCondition$Context): Argument
leaf type org.apache.flink.cep.pattern.conditions.IterativeCondition$Context
does not satisfy: reside outside of package 'org.apache.flink..' or reside in
any package ['..shaded..'] or annotated with @Public or annotated with
@PublicEvolving or annotated with @Deprecated
{code}
This doesn't make any sense given that {{IterativeCondition}} itself is already
{{PublicEvolving}} and contains the exact same method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29413) Make it possible to associate triggered and completed savepoints
[ https://issues.apache.org/jira/browse/FLINK-29413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-29413: --- Fix Version/s: kubernetes-operator-1.3.0 > Make it possible to associate triggered and completed savepoints > > > Key: FLINK-29413 > URL: https://issues.apache.org/jira/browse/FLINK-29413 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Major > Fix For: kubernetes-operator-1.3.0 > > > Currently it is not clear how one would assoicate completed manual savepoints > with savepointTriggerNonce-es when using the operator. > This makes it difficult to track when a savepoint was completed vs when it > was abandoned > One idea would be to add the savepointTriggerNonce to the completed > checkpoint info for Manual savepoints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-29413) Make it possible to associate triggered and completed savepoints
[ https://issues.apache.org/jira/browse/FLINK-29413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora reassigned FLINK-29413: -- Assignee: Gyula Fora > Make it possible to associate triggered and completed savepoints > > > Key: FLINK-29413 > URL: https://issues.apache.org/jira/browse/FLINK-29413 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Major > > Currently it is not clear how one would assoicate completed manual savepoints > with savepointTriggerNonce-es when using the operator. > This makes it difficult to track when a savepoint was completed vs when it > was abandoned > One idea would be to add the savepointTriggerNonce to the completed > checkpoint info for Manual savepoints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29413) Make it possible to associate triggered and completed savepoints
[ https://issues.apache.org/jira/browse/FLINK-29413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609572#comment-17609572 ] Gyula Fora commented on FLINK-29413: We don't really see any other way to retrieve the format info, so it is probably valuable to track that also > Make it possible to associate triggered and completed savepoints > > > Key: FLINK-29413 > URL: https://issues.apache.org/jira/browse/FLINK-29413 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Currently it is not clear how one would assoicate completed manual savepoints > with savepointTriggerNonce-es when using the operator. > This makes it difficult to track when a savepoint was completed vs when it > was abandoned > One idea would be to add the savepointTriggerNonce to the completed > checkpoint info for Manual savepoints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29413) Make it possible to associate triggered and completed savepoints
[ https://issues.apache.org/jira/browse/FLINK-29413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609561#comment-17609561 ] Thomas Weise commented on FLINK-29413: -- +1 for tracking the triggering nonce Would there be any other way to retrieve the savepoint type given that Flink does not retain a history beyond current job execution? > Make it possible to associate triggered and completed savepoints > > > Key: FLINK-29413 > URL: https://issues.apache.org/jira/browse/FLINK-29413 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Currently it is not clear how one would assoicate completed manual savepoints > with savepointTriggerNonce-es when using the operator. > This makes it difficult to track when a savepoint was completed vs when it > was abandoned > One idea would be to add the savepointTriggerNonce to the completed > checkpoint info for Manual savepoints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol commented on a diff in pull request #20893: [FLINK-29403][cep] Streamline SimpleCondition usage
zentol commented on code in PR #20893:
URL: https://github.com/apache/flink/pull/20893#discussion_r980213947
##
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/pattern/conditions/SimpleCondition.java:
##
@@ -40,4 +40,13 @@
public boolean filter(T value, Context ctx) throws Exception {
return filter(value);
}
+
+public static SimpleCondition of(FilterFunction filters) {
Review Comment:
> replace the `Condition` with the `FilterFunction`
I'm not sure I understand. Which `Condition` to do you want to _replace_?
> other condition implementation should follow this usage
I'm not sure this is needed; AFAICT all other IterativeCondition sub-classes
are truly internal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gaborgsomogyi commented on pull request #386: [FLINK-29376] Set SHUTDOWN_ON_APPLICATION_FINISH to false only on 1.15 and above
gaborgsomogyi commented on PR #386: URL: https://github.com/apache/flink-kubernetes-operator/pull/386#issuecomment-1258232704 cc @gyfora -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29376) Deployment already exists error if Flink version is not set correctly
[ https://issues.apache.org/jira/browse/FLINK-29376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29376: --- Labels: pull-request-available (was: ) > Deployment already exists error if Flink version is not set correctly > - > > Key: FLINK-29376 > URL: https://issues.apache.org/jira/browse/FLINK-29376 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.1.0, kubernetes-operator-1.2.0 >Reporter: Gyula Fora >Assignee: Gabor Somogyi >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.2.0 > > > If the user incorrectly sets a Flink version lower than 1.15 when using 1.15 > or above there are some strange behaviour currently around cluster shutdown. > Since we always set SHUTDOWN_ON_APPLICATION_FINISH (in the > FlinkConfigBuilder) regardless of version but set the JmDeployStatus to > MISSING/READY based on version. It can happen that we set the JmStatus to > MISSING when the jm deployment is still running. > As the observer skips observing UPGRADING CRs it never updates the MISSING > status and the deployment logic fails on duplicate deployment. > An easy fix could be to only set SHUTDOWN_ON_APPLICATION_FINISH for version > >= 1.15 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] gaborgsomogyi opened a new pull request, #386: [FLINK-29376] Set SHUTDOWN_ON_APPLICATION_FINISH to false only on 1.15 and above
gaborgsomogyi opened a new pull request, #386: URL: https://github.com/apache/flink-kubernetes-operator/pull/386 ## What is the purpose of the change If the user incorrectly sets a Flink version lower than 1.15 when using 1.15 or above: ``` image: flink:1.15 flinkVersion: v1_14 ``` then a spec change ends-up in exception and restart loop. In this PR we set `SHUTDOWN_ON_APPLICATION_FINISH` to `false` when Flink version is less than 1.15. ## Brief change log * Set `SHUTDOWN_ON_APPLICATION_FINISH` to false only on 1.15 and above * Added unit test for all available Flink versions ## Verifying this change * New unit test * Manually * Deploy app where `image: flink:1.15 flinkVersion: v1_14` * Wait for RUNNING state * Change Flink config * Double check that Flink is re-deployed and in RUNNING state again ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changes to the `CustomResourceDescriptors`: no - Core observer or reconciler logic that is regularly executed: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gaborgsomogyi commented on a diff in pull request #382: [FLINK-29392] Trigger error when session job is lost without HA
gaborgsomogyi commented on code in PR #382:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/382#discussion_r980187297
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/JobStatusObserver.java:
##
@@ -72,22 +81,91 @@ public boolean observe(
}
if (!clusterJobStatuses.isEmpty()) {
+// There are jobs on the cluster, we filter the ones for this
resource
Optional targetJobStatusMessage =
filterTargetJob(jobStatus, clusterJobStatuses);
+
if (targetJobStatusMessage.isEmpty()) {
-
jobStatus.setState(org.apache.flink.api.common.JobStatus.RECONCILING.name());
+LOG.warn("No matching jobs found on the cluster");
+ifRunningMoveToReconciling(jobStatus, previousJobStatus);
+// We could list the jobs but cannot find the one for this
resource
+if (resource instanceof FlinkDeployment) {
+// This should never happen for application clusters,
there is something wrong
+setUnknownJobError((FlinkDeployment) resource);
+} else {
+ifHaDisabledMarkSessionJobMissing((FlinkSessionJob)
resource, deployedConfig);
+}
return false;
} else {
updateJobStatus(resource, targetJobStatusMessage.get(),
deployedConfig);
}
ReconciliationUtils.checkAndUpdateStableSpec(resource.getStatus());
return true;
} else {
+LOG.debug("No jobs found on the cluster");
+// No jobs found on the cluster, it is possible that the
jobmanager is still starting up
ifRunningMoveToReconciling(jobStatus, previousJobStatus);
+
+if (resource instanceof FlinkSessionJob) {
Review Comment:
Seems like this is not common logic, maybe this can be extracted from here.
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/JobStatusObserver.java:
##
@@ -72,22 +81,91 @@ public boolean observe(
}
if (!clusterJobStatuses.isEmpty()) {
+// There are jobs on the cluster, we filter the ones for this
resource
Optional targetJobStatusMessage =
filterTargetJob(jobStatus, clusterJobStatuses);
+
if (targetJobStatusMessage.isEmpty()) {
-
jobStatus.setState(org.apache.flink.api.common.JobStatus.RECONCILING.name());
+LOG.warn("No matching jobs found on the cluster");
+ifRunningMoveToReconciling(jobStatus, previousJobStatus);
+// We could list the jobs but cannot find the one for this
resource
+if (resource instanceof FlinkDeployment) {
+// This should never happen for application clusters,
there is something wrong
+setUnknownJobError((FlinkDeployment) resource);
+} else {
+ifHaDisabledMarkSessionJobMissing((FlinkSessionJob)
resource, deployedConfig);
+}
return false;
} else {
updateJobStatus(resource, targetJobStatusMessage.get(),
deployedConfig);
}
ReconciliationUtils.checkAndUpdateStableSpec(resource.getStatus());
return true;
} else {
+LOG.debug("No jobs found on the cluster");
+// No jobs found on the cluster, it is possible that the
jobmanager is still starting up
ifRunningMoveToReconciling(jobStatus, previousJobStatus);
+
+if (resource instanceof FlinkSessionJob) {
+ifHaDisabledMarkSessionJobMissing((FlinkSessionJob) resource,
deployedConfig);
+}
return false;
}
}
+/**
+ * When HA is disabled the session job will not recover on JM restarts. If
the JM goes down /
+ * restarted the session job should be marked missing.
+ *
+ * @param sessionJob Flink session job.
+ * @param conf Flink config.
+ */
+private void ifHaDisabledMarkSessionJobMissing(FlinkSessionJob sessionJob,
Configuration conf) {
Review Comment:
This is more like a taste question so we may or may not do it. I prefer
`get/is` and `set` functions and build upper level functionality like
```
if (something) {
react
}
```
More re-usable if somebody wants to implement
`ifTheMoonIsGreenMarkSessionJobMissing` functionality.
But again, this is taste question so we can leave it like it if you don't
agree.
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/observer/JobStatusObserver.java:
##
@@ -72,22 +81,91 @@ public boolean observe(
}
if (!clusterJobStatuses.isEmpty()) {
+// There are jobs on the cluster, we filter the
[jira] [Closed] (FLINK-29109) Checkpoint path conflict with stateless upgrade mode
[ https://issues.apache.org/jira/browse/FLINK-29109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-29109. -- Fix Version/s: kubernetes-operator-1.2.0 Resolution: Fixed merged to main 6f0914bbd296c9daf2664afe0a77d1df4f2e157e > Checkpoint path conflict with stateless upgrade mode > > > Key: FLINK-29109 > URL: https://issues.apache.org/jira/browse/FLINK-29109 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Affects Versions: kubernetes-operator-1.1.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.2.0 > > > A stateful job with stateless upgrade mode (yes, there are such use cases) > fails with checkpoint path conflict due to constant jobId and FLINK-19358 > (applies to Flink < 1.16x). Since with stateless upgrade mode the checkpoint > id resets on restart the job is going to write to previously used locations > and fail. The workaround is to rotate the jobId on every redeploy when the > upgrade mode is stateless. While this can be worked around externally it is > best done in the operator itself because reconciliation resolves when a > restart is actually required while rotating jobId externally may trigger > unnecessary restarts. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29413) Make it possible to associate triggered and completed savepoints
[ https://issues.apache.org/jira/browse/FLINK-29413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609537#comment-17609537 ] Gyula Fora commented on FLINK-29413: I guess we could add that too if there is no other way to get that info, [~thw] what do you think? > Make it possible to associate triggered and completed savepoints > > > Key: FLINK-29413 > URL: https://issues.apache.org/jira/browse/FLINK-29413 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Currently it is not clear how one would assoicate completed manual savepoints > with savepointTriggerNonce-es when using the operator. > This makes it difficult to track when a savepoint was completed vs when it > was abandoned > One idea would be to add the savepointTriggerNonce to the completed > checkpoint info for Manual savepoints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29413) Make it possible to associate triggered and completed savepoints
[ https://issues.apache.org/jira/browse/FLINK-29413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609527#comment-17609527 ] Matyas Orhidi commented on FLINK-29413: --- Would it makes sense to add the format type too, for all cases until https://issues.apache.org/jira/browse/FLINK-29322 is fixed? > Make it possible to associate triggered and completed savepoints > > > Key: FLINK-29413 > URL: https://issues.apache.org/jira/browse/FLINK-29413 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > > Currently it is not clear how one would assoicate completed manual savepoints > with savepointTriggerNonce-es when using the operator. > This makes it difficult to track when a savepoint was completed vs when it > was abandoned > One idea would be to add the savepointTriggerNonce to the completed > checkpoint info for Manual savepoints. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] mbalassi commented on a diff in pull request #383: [FLINK-29384] Bump snakeyaml version to 1.32
mbalassi commented on code in PR #383:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/383#discussion_r980116122
##
flink-kubernetes-operator/pom.xml:
##
@@ -47,6 +47,20 @@ under the License.
io.fabric8
kubernetes-client
${fabric8.version}
+
+
+
+org.yaml
+snakeyaml
+
+
+
+
+
+
Review Comment:
I tried the dependency management route in `flink-kubernetes-parent`, it did
not seems to work. I see your point, and can give it a try if you prefer. Given
that it is hopefully just a temporary fix I would suggest to leave it as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-29421) Support python 3.10
Eric Sirianni created FLINK-29421:
-
Summary: Support python 3.10
Key: FLINK-29421
URL: https://issues.apache.org/jira/browse/FLINK-29421
Project: Flink
Issue Type: Bug
Components: API / Python
Reporter: Eric Sirianni
The {{apache-flink}} package fails to install on Python 3.10 due to inability
to compile {{numpy}}
{noformat}
numpy/core/src/multiarray/scalartypes.c.src:3242:12: error: too few
arguments to function ‘_Py_HashDouble’
3242 | return
_Py_HashDouble(npy_half_to_double(((PyHalfScalarObject *)obj)->obval));
|^~
In file included from
/home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/Python.h:77,
from numpy/core/src/multiarray/scalartypes.c.src:3:
/home/sirianni/.asdf/installs/python/3.10.6/include/python3.10/pyhash.h:10:23:
note: declared here
10 | PyAPI_FUNC(Py_hash_t) _Py_HashDouble(PyObject *, double);
{noformat}
Numpy issue https://github.com/numpy/numpy/issues/19033
[Mailing list
thread|https://lists.apache.org/thread/f4r9hjt1l33xf5ngnswszhnls4cxkk52]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] mbalassi commented on a diff in pull request #383: [FLINK-29384] Bump snakeyaml version to 1.32
mbalassi commented on code in PR #383: URL: https://github.com/apache/flink-kubernetes-operator/pull/383#discussion_r980109419 ## flink-kubernetes-shaded/pom.xml: ## @@ -88,6 +103,7 @@ under the License. META-INF/DEPENDENCIES META-INF/LICENSE META-INF/MANIFEST.MF + org/apache/flink/kubernetes/shaded/org/yaml/snakeyaml/** Review Comment: @wangyang0918 The NOTICE has this and only this relevant entry: ``` - SnakeYAML (https://bitbucket.org/snakeyaml/snakeyaml) org.yaml:snakeyaml:bundle:1.32 License: Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0.txt) ``` @gyfora Because of the relocation, I added comments to make it more clear. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29420) Add support for Zookeeper 3.7 / 3.8
Chesnay Schepler created FLINK-29420: Summary: Add support for Zookeeper 3.7 / 3.8 Key: FLINK-29420 URL: https://issues.apache.org/jira/browse/FLINK-29420 Project: Flink Issue Type: Improvement Components: BuildSystem / Shaded, Runtime / Coordination Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: 1.17.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] tweise merged pull request #385: [FLINK-29109] Avoid checkpoint path conflicts (Flink < 1.16)
tweise merged PR #385: URL: https://github.com/apache/flink-kubernetes-operator/pull/385 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29414) Setup license checks
[ https://issues.apache.org/jira/browse/FLINK-29414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-29414. Resolution: Fixed master: 575912de77153544a5e4211f504cc67e5ee776a8 > Setup license checks > > > Key: FLINK-29414 > URL: https://issues.apache.org/jira/browse/FLINK-29414 > Project: Flink > Issue Type: Technical Debt > Components: BuildSystem / Shaded >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: shaded-1.16 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-shaded] zentol merged pull request #111: [FLINK-29414] Setup license check
zentol merged PR #111: URL: https://github.com/apache/flink-shaded/pull/111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29416) One or more fetchers have encountered exception
[
https://issues.apache.org/jira/browse/FLINK-29416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609511#comment-17609511
]
Weijie Guo commented on FLINK-29416:
It seems that direct memory is not enough, you can try to increase the size of
taskmanager.memory.task.off-heap.size
> One or more fetchers have encountered exception
> ---
>
> Key: FLINK-29416
> URL: https://issues.apache.org/jira/browse/FLINK-29416
> Project: Flink
> Issue Type: Bug
>Reporter: liuchenhong
>Priority: Major
>
> One machine in the cluster goes offline, most tasks failover, and then it
> automatically restarts successfully. But the task of writing data to kudu
> failed to restart. The error is as follows:
>
> {code:java}
> //代码占位符
> 2022-09-26 16:06:30,634 WARN org.apache.flink.runtime.taskmanager.Task
> [] - KunlunRecLogOutputWindow (11/90)#6
> (9e23189bdc1ba38b19f922342819dab2) switched from RUNNING to FAILED with
> failure cause:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Error at remote task manager 'bjstream44.jd.163.org/10.196.24.76:39604'.
> at
> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.decodeMsg(CreditBasedPartitionRequestClientHandler.java:351)
> at
> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelRead(CreditBasedPartitionRequestClientHandler.java:240)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelRead(NettyMessageClientDecoderDelegate.java:112)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.flink.runtime.io.network.partition.ProducerFailedException:
> java.lang.RuntimeException: One or more fetchers have encountered exception
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.writeAndFlushNextMessageIfPossible(PartitionRequestQueue.java:285)
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.enqueueAvailableReader(PartitionRequestQueue.java:123)
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.userEventTriggered(PartitionRequestQueue.java:234)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
> at
>
[GitHub] [flink-shaded] zentol commented on a diff in pull request #111: [FLINK-29414] Setup license check
zentol commented on code in PR #111: URL: https://github.com/apache/flink-shaded/pull/111#discussion_r980060247 ## .github/workflows/ci.yml: ## @@ -5,12 +5,37 @@ on: [push, pull_request] jobs: build: runs-on: ubuntu-latest +env: + MVN_COMMON_OPTIONS: -U -B --no-transfer-progress Review Comment: Yes. AFAIK those are workarounds for older maven versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-shaded] MartijnVisser commented on a diff in pull request #111: [FLINK-29414] Setup license check
MartijnVisser commented on code in PR #111: URL: https://github.com/apache/flink-shaded/pull/111#discussion_r980050857 ## .github/workflows/ci.yml: ## @@ -5,12 +5,37 @@ on: [push, pull_request] jobs: build: runs-on: ubuntu-latest +env: + MVN_COMMON_OPTIONS: -U -B --no-transfer-progress Review Comment: Did you deliberately omit `MVN_CONNECTION_OPTIONS` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #20902: [FLINK-29419] fix the problem that testHybridFullExchangesRestart hang.
flinkbot commented on PR #20902: URL: https://github.com/apache/flink/pull/20902#issuecomment-1258054514 ## CI report: * f57a254ef0d910a65628bb709d8030cf7a63f39b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29419) HybridShuffle.testHybridFullExchangesRestart hangs
[
https://issues.apache.org/jira/browse/FLINK-29419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-29419:
---
Labels: pull-request-available test-stability (was: test-stability)
> HybridShuffle.testHybridFullExchangesRestart hangs
> --
>
> Key: FLINK-29419
> URL: https://issues.apache.org/jira/browse/FLINK-29419
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {code:java}
> 2022-09-26T10:56:44.0766792Z Sep 26 10:56:44 "ForkJoinPool-1-worker-25" #27
> daemon prio=5 os_prio=0 tid=0x7f41a4efa000 nid=0x6d76 waiting on
> condition [0x7f40ac135000]
> 2022-09-26T10:56:44.0767432Z Sep 26 10:56:44java.lang.Thread.State:
> WAITING (parking)
> 2022-09-26T10:56:44.0767892Z Sep 26 10:56:44 at sun.misc.Unsafe.park(Native
> Method)
> 2022-09-26T10:56:44.0768644Z Sep 26 10:56:44 - parking to wait for
> <0xa0704e18> (a java.util.concurrent.CompletableFuture$Signaller)
> 2022-09-26T10:56:44.0769287Z Sep 26 10:56:44 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2022-09-26T10:56:44.0769949Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2022-09-26T10:56:44.0770623Z Sep 26 10:56:44 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> 2022-09-26T10:56:44.0771349Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2022-09-26T10:56:44.0772092Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-09-26T10:56:44.0772777Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:57)
> 2022-09-26T10:56:44.0773534Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:115)
> 2022-09-26T10:56:44.0774333Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridFullExchangesRestart(HybridShuffleITCase.java:59)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41343=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] reswqa opened a new pull request, #20902: [FLINK-29419] fix the problem that testHybridFullExchangesRestart hang.
reswqa opened a new pull request, #20902: URL: https://github.com/apache/flink/pull/20902 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29415) InitializationFailure when recovering from a checkpoint in Application Mode leads to the cleanup of all HA data
[
https://issues.apache.org/jira/browse/FLINK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl closed FLINK-29415.
-
Resolution: Invalid
There is a solution for this specific case already with
{{execution.submit-failed-job-on-application-error=true}} being introduced with
FLINK-25715 in Flink 1.15 as [~gyfora] pointed out in the ML thread. I'm gonna
close this issue again.
> InitializationFailure when recovering from a checkpoint in Application Mode
> leads to the cleanup of all HA data
> ---
>
> Key: FLINK-29415
> URL: https://issues.apache.org/jira/browse/FLINK-29415
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.16.0, 1.17.0, 1.15.2, 1.14.6
>Reporter: Matthias Pohl
>Priority: Major
>
> This issue was raised in the user ML thread [JobManager restarts on job
> failure|https://lists.apache.org/thread/qkmcty3h4gkkx5g09m19gwqrf8z8d383].
> Recovering from a external checkpoint is handled differently than recovering
> from an internal state (see
> [Dispatcher#handleJobManagerRunner|https://github.com/apache/flink/blob/41ac1ba13679121f1ddf14b26a36f4f4a3cc73e4/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L651]).
> For the latter case, we explicitly do a local cleanup (i.e. no HA data is
> cleaned up). For the case, described in the ML thread, a global cleanup is
> performed. That's not a problem in session mode where a new job ID is used.
> The new job ID will result in using a new namespace for the HA data. Data
> from previous runs are not touched during a cleanup. In Application mode, we
> use the default job ID `0` which would be reused. In case of a failure, all
> the HA data will be "namespaced" using the default job id. As a consequence,
> the related data is cleaned up.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29412) Connection leak in orc reader
[ https://issues.apache.org/jira/browse/FLINK-29412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-29412. Resolution: Fixed master: 29fc9adf023a04b02126f50d915feb55fdbc2327 release-0.2: 311ab67dd2d7dcb4a7a18bff1de6a2da5861964e > Connection leak in orc reader > - > > Key: FLINK-29412 > URL: https://issues.apache.org/jira/browse/FLINK-29412 > Project: Flink > Issue Type: Bug > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Blocker > Labels: pull-request-available > Fix For: table-store-0.3.0, table-store-0.2.1 > > > 1. OrcFileStatsExtractor forget closing reader. > 2. HadoopReadOnlyFileSystem forget closing fsDataInputStream. > We need a pocket test to assert all connections are closed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29419) HybridShuffle.testHybridFullExchangesRestart hangs
[
https://issues.apache.org/jira/browse/FLINK-29419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609486#comment-17609486
]
Weijie Guo commented on FLINK-29419:
[~hxbks2ks] Thanks for the feedback. I will check the cause of this problem.
> HybridShuffle.testHybridFullExchangesRestart hangs
> --
>
> Key: FLINK-29419
> URL: https://issues.apache.org/jira/browse/FLINK-29419
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-26T10:56:44.0766792Z Sep 26 10:56:44 "ForkJoinPool-1-worker-25" #27
> daemon prio=5 os_prio=0 tid=0x7f41a4efa000 nid=0x6d76 waiting on
> condition [0x7f40ac135000]
> 2022-09-26T10:56:44.0767432Z Sep 26 10:56:44java.lang.Thread.State:
> WAITING (parking)
> 2022-09-26T10:56:44.0767892Z Sep 26 10:56:44 at sun.misc.Unsafe.park(Native
> Method)
> 2022-09-26T10:56:44.0768644Z Sep 26 10:56:44 - parking to wait for
> <0xa0704e18> (a java.util.concurrent.CompletableFuture$Signaller)
> 2022-09-26T10:56:44.0769287Z Sep 26 10:56:44 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2022-09-26T10:56:44.0769949Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2022-09-26T10:56:44.0770623Z Sep 26 10:56:44 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> 2022-09-26T10:56:44.0771349Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2022-09-26T10:56:44.0772092Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-09-26T10:56:44.0772777Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:57)
> 2022-09-26T10:56:44.0773534Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:115)
> 2022-09-26T10:56:44.0774333Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridFullExchangesRestart(HybridShuffleITCase.java:59)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41343=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #304: [FLINK-29412] Connection leak in orc reader
JingsongLi merged PR #304: URL: https://github.com/apache/flink-table-store/pull/304 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-29419) HybridShuffle.testHybridFullExchangesRestart hangs
[
https://issues.apache.org/jira/browse/FLINK-29419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609482#comment-17609482
]
Huang Xingbo commented on FLINK-29419:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41345=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba
> HybridShuffle.testHybridFullExchangesRestart hangs
> --
>
> Key: FLINK-29419
> URL: https://issues.apache.org/jira/browse/FLINK-29419
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-26T10:56:44.0766792Z Sep 26 10:56:44 "ForkJoinPool-1-worker-25" #27
> daemon prio=5 os_prio=0 tid=0x7f41a4efa000 nid=0x6d76 waiting on
> condition [0x7f40ac135000]
> 2022-09-26T10:56:44.0767432Z Sep 26 10:56:44java.lang.Thread.State:
> WAITING (parking)
> 2022-09-26T10:56:44.0767892Z Sep 26 10:56:44 at sun.misc.Unsafe.park(Native
> Method)
> 2022-09-26T10:56:44.0768644Z Sep 26 10:56:44 - parking to wait for
> <0xa0704e18> (a java.util.concurrent.CompletableFuture$Signaller)
> 2022-09-26T10:56:44.0769287Z Sep 26 10:56:44 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2022-09-26T10:56:44.0769949Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2022-09-26T10:56:44.0770623Z Sep 26 10:56:44 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> 2022-09-26T10:56:44.0771349Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2022-09-26T10:56:44.0772092Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-09-26T10:56:44.0772777Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:57)
> 2022-09-26T10:56:44.0773534Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:115)
> 2022-09-26T10:56:44.0774333Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridFullExchangesRestart(HybridShuffleITCase.java:59)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41343=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29419) HybridShuffle.testHybridFullExchangesRestart hangs
[
https://issues.apache.org/jira/browse/FLINK-29419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609481#comment-17609481
]
Huang Xingbo commented on FLINK-29419:
--
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41345=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
> HybridShuffle.testHybridFullExchangesRestart hangs
> --
>
> Key: FLINK-29419
> URL: https://issues.apache.org/jira/browse/FLINK-29419
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-26T10:56:44.0766792Z Sep 26 10:56:44 "ForkJoinPool-1-worker-25" #27
> daemon prio=5 os_prio=0 tid=0x7f41a4efa000 nid=0x6d76 waiting on
> condition [0x7f40ac135000]
> 2022-09-26T10:56:44.0767432Z Sep 26 10:56:44java.lang.Thread.State:
> WAITING (parking)
> 2022-09-26T10:56:44.0767892Z Sep 26 10:56:44 at sun.misc.Unsafe.park(Native
> Method)
> 2022-09-26T10:56:44.0768644Z Sep 26 10:56:44 - parking to wait for
> <0xa0704e18> (a java.util.concurrent.CompletableFuture$Signaller)
> 2022-09-26T10:56:44.0769287Z Sep 26 10:56:44 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2022-09-26T10:56:44.0769949Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2022-09-26T10:56:44.0770623Z Sep 26 10:56:44 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> 2022-09-26T10:56:44.0771349Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2022-09-26T10:56:44.0772092Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-09-26T10:56:44.0772777Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:57)
> 2022-09-26T10:56:44.0773534Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:115)
> 2022-09-26T10:56:44.0774333Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridFullExchangesRestart(HybridShuffleITCase.java:59)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41343=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29419) HybridShuffle.testHybridFullExchangesRestart hangs
[
https://issues.apache.org/jira/browse/FLINK-29419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609480#comment-17609480
]
Huang Xingbo commented on FLINK-29419:
--
cc [~Weijie Guo]
> HybridShuffle.testHybridFullExchangesRestart hangs
> --
>
> Key: FLINK-29419
> URL: https://issues.apache.org/jira/browse/FLINK-29419
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0
>Reporter: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-26T10:56:44.0766792Z Sep 26 10:56:44 "ForkJoinPool-1-worker-25" #27
> daemon prio=5 os_prio=0 tid=0x7f41a4efa000 nid=0x6d76 waiting on
> condition [0x7f40ac135000]
> 2022-09-26T10:56:44.0767432Z Sep 26 10:56:44java.lang.Thread.State:
> WAITING (parking)
> 2022-09-26T10:56:44.0767892Z Sep 26 10:56:44 at sun.misc.Unsafe.park(Native
> Method)
> 2022-09-26T10:56:44.0768644Z Sep 26 10:56:44 - parking to wait for
> <0xa0704e18> (a java.util.concurrent.CompletableFuture$Signaller)
> 2022-09-26T10:56:44.0769287Z Sep 26 10:56:44 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 2022-09-26T10:56:44.0769949Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> 2022-09-26T10:56:44.0770623Z Sep 26 10:56:44 at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
> 2022-09-26T10:56:44.0771349Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> 2022-09-26T10:56:44.0772092Z Sep 26 10:56:44 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-09-26T10:56:44.0772777Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:57)
> 2022-09-26T10:56:44.0773534Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:115)
> 2022-09-26T10:56:44.0774333Z Sep 26 10:56:44 at
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridFullExchangesRestart(HybridShuffleITCase.java:59)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41343=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-27346) [umbrella] Introduce Hive reader for table store
[ https://issues.apache.org/jira/browse/FLINK-27346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-27346. Resolution: Fixed > [umbrella] Introduce Hive reader for table store > > > Key: FLINK-27346 > URL: https://issues.apache.org/jira/browse/FLINK-27346 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jingsong Lee >Assignee: Caizhi Weng >Priority: Minor > Fix For: table-store-0.3.0 > > > Add the ability of Hive engine to read table of Flink Table Store. > Currently the typical query engine is still the Hive engine. For storage, the > ecology of queries is very important. > We currently write an InputFormat for Hive, which requires us to: > - Organize code dependencies to avoid creating dependency conflicts with Hive > - some necessary refactoring to make the programming interface more generic > - Write the Hive InputFormat and implement some pushdown -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-29419) HybridShuffle.testHybridFullExchangesRestart hangs
Huang Xingbo created FLINK-29419:
Summary: HybridShuffle.testHybridFullExchangesRestart hangs
Key: FLINK-29419
URL: https://issues.apache.org/jira/browse/FLINK-29419
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.17.0
Reporter: Huang Xingbo
{code:java}
2022-09-26T10:56:44.0766792Z Sep 26 10:56:44 "ForkJoinPool-1-worker-25" #27
daemon prio=5 os_prio=0 tid=0x7f41a4efa000 nid=0x6d76 waiting on condition
[0x7f40ac135000]
2022-09-26T10:56:44.0767432Z Sep 26 10:56:44java.lang.Thread.State: WAITING
(parking)
2022-09-26T10:56:44.0767892Z Sep 26 10:56:44at sun.misc.Unsafe.park(Native
Method)
2022-09-26T10:56:44.0768644Z Sep 26 10:56:44- parking to wait for
<0xa0704e18> (a java.util.concurrent.CompletableFuture$Signaller)
2022-09-26T10:56:44.0769287Z Sep 26 10:56:44at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
2022-09-26T10:56:44.0769949Z Sep 26 10:56:44at
java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
2022-09-26T10:56:44.0770623Z Sep 26 10:56:44at
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
2022-09-26T10:56:44.0771349Z Sep 26 10:56:44at
java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
2022-09-26T10:56:44.0772092Z Sep 26 10:56:44at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
2022-09-26T10:56:44.0772777Z Sep 26 10:56:44at
org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:57)
2022-09-26T10:56:44.0773534Z Sep 26 10:56:44at
org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:115)
2022-09-26T10:56:44.0774333Z Sep 26 10:56:44at
org.apache.flink.test.runtime.HybridShuffleITCase.testHybridFullExchangesRestart(HybridShuffleITCase.java:59)
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=41343=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27843) Schema evolution for data file meta
[ https://issues.apache.org/jira/browse/FLINK-27843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609470#comment-17609470 ] liwei li commented on FLINK-27843: -- May i have a try? > Schema evolution for data file meta > --- > > Key: FLINK-27843 > URL: https://issues.apache.org/jira/browse/FLINK-27843 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Reporter: Jingsong Lee >Priority: Minor > Fix For: table-store-0.3.0 > > > There are quite a few metadata operations on DataFileMeta, such as getting > the statistics of each column and the partition of the file. > We need to evolution to the latest schema based on schemaId when we get this > information -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29243) Introduce a COW mode for table store
[ https://issues.apache.org/jira/browse/FLINK-29243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609465#comment-17609465 ] liwei li commented on FLINK-29243: -- i want to have a try, could you gave me the ticket? > Introduce a COW mode for table store > > > Key: FLINK-29243 > URL: https://issues.apache.org/jira/browse/FLINK-29243 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Priority: Major > Fix For: table-store-0.3.0 > > > Currently, we can combine the COW (Copy on write) mode by three options, but > the combination may not be optimal and tedious. > We can introduce a option that specifically turns on the mode of COW. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #20901: [FLINK-26943][table-planner-blink] Support date_add function in sql
flinkbot commented on PR #20901: URL: https://github.com/apache/flink/pull/20901#issuecomment-1257928071 ## CI report: * e9ed9084724236b500271fdcf5d1de5aef056496 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-29418) Update flink-shaded
Chesnay Schepler created FLINK-29418: Summary: Update flink-shaded Key: FLINK-29418 URL: https://issues.apache.org/jira/browse/FLINK-29418 Project: Flink Issue Type: Technical Debt Components: BuildSystem / Shaded Reporter: Chesnay Schepler Assignee: Chesnay Schepler Fix For: shaded-1.16 Do a round of updates in flink-shaded. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] y0908105023 commented on pull request #20846: [FLINK-26943] [add date_add function support]
y0908105023 commented on PR #20846: URL: https://github.com/apache/flink/pull/20846#issuecomment-1257921682 @snuyanzin i request a new pr: https://github.com/apache/flink/pull/20901. thaks for your time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] y0908105023 commented on pull request #20901: [FLINK-26943][table-planner-blink] Support date_add function in sql
y0908105023 commented on PR #20901: URL: https://github.com/apache/flink/pull/20901#issuecomment-1257920212 history pr： https://github.com/apache/flink/pull/20846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-shaded] zentol commented on a diff in pull request #111: [FLINK-29414] Setup license check
zentol commented on code in PR #111:
URL: https://github.com/apache/flink-shaded/pull/111#discussion_r979915416
##
pom.xml:
##
@@ -51,9 +51,23 @@ under the License.
scm:git:https://github.com/apache/flink-shaded.git
+
+
+
+apache.snapshots.https
+${distMgmtSnapshotsName}
+${distMgmtSnapshotsUrl}
+
+
+
org.apache.flink.shaded
4.1.70.Final
+
[GitHub] [flink] y0908105023 opened a new pull request, #20901: [FLINK-26943][table-planner-blink] Support date_add function in sql
y0908105023 opened a new pull request, #20901: URL: https://github.com/apache/flink/pull/20901 What is the purpose of the change Add date_add sql function support Brief change log date_add add sql function date_add to BuiltInFunction Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing This change is already covered by existing tests, such as (please describe tests). ScalarFunctionsTest#testDateTypeAdd ScalarFunctionsTest#testDateTypeMinus Does this pull request potentially affect one of the following parts: Dependencies (does it add or upgrade a dependency): (yes / no) The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no) The serializers: (yes / no / don't know) The runtime per-record code paths (performance sensitive): (yes / no / don't know) Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) The S3 file system connector: (yes / no / don't know) It does not affect any parts Documentation Does this pull request introduce a new feature? (yes / no) If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-shaded] zentol commented on a diff in pull request #111: [FLINK-29414] Setup license check
zentol commented on code in PR #111:
URL: https://github.com/apache/flink-shaded/pull/111#discussion_r979911314
##
pom.xml:
##
@@ -51,9 +51,23 @@ under the License.
scm:git:https://github.com/apache/flink-shaded.git
+
+
+
+apache.snapshots.https
+${distMgmtSnapshotsName}
+${distMgmtSnapshotsUrl}
+
+
+
org.apache.flink.shaded
4.1.70.Final
+
+
Review Comment:
This thing is super jank but I don't have an idea on how to solve it in
_general_ without potentially breaking _something_ (because we have to either
_add_ or _remove_ suffixes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-shaded] zentol commented on a diff in pull request #111: [FLINK-29414] Setup license check
zentol commented on code in PR #111:
URL: https://github.com/apache/flink-shaded/pull/111#discussion_r979911314
##
pom.xml:
##
@@ -51,9 +51,23 @@ under the License.
scm:git:https://github.com/apache/flink-shaded.git
+
+
+
+apache.snapshots.https
+${distMgmtSnapshotsName}
+${distMgmtSnapshotsUrl}
+
+
+
org.apache.flink.shaded
4.1.70.Final
+
+
Review Comment:
This thing is super jank but I don't have an idea on how to solve it in
_general_ without potentially breaking _something_.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29414) Setup license checks
[ https://issues.apache.org/jira/browse/FLINK-29414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29414: --- Labels: pull-request-available (was: ) > Setup license checks > > > Key: FLINK-29414 > URL: https://issues.apache.org/jira/browse/FLINK-29414 > Project: Flink > Issue Type: Technical Debt > Components: BuildSystem / Shaded >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: shaded-1.16 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] y0908105023 closed pull request #20846: [FLINK-26943] [add date_add function support]
y0908105023 closed pull request #20846: [FLINK-26943] [add date_add function support] URL: https://github.com/apache/flink/pull/20846 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-shaded] zentol opened a new pull request, #111: [FLINK-29414] Setup license check
zentol opened a new pull request, #111: URL: https://github.com/apache/flink-shaded/pull/111 Most of the changes are taken 1:1 from the ES repo. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] y0908105023 commented on a diff in pull request #20846: [FLINK-26943] [add date_add function support]
y0908105023 commented on code in PR #20846:
URL: https://github.com/apache/flink/pull/20846#discussion_r979900275
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/DateTimeUtils.java:
##
@@ -1400,6 +1401,25 @@ public static String formatUnixTimestamp(long unixtime,
TimeZone tz) {
return formatUnixTimestamp(unixtime, TIMESTAMP_FORMAT_STRING, tz);
}
+/** Add offset to string (like '1970-01-01') and return new date For
String support. */
+public static Integer dateAdd(BinaryStringData binaryStringData, int
offset) {
+
+if (binaryStringData == null) {
+return null;
+}
+try {
+int days = parseDate(binaryStringData.toString());
+return days + offset;
+} catch (NullPointerException e) {
Review Comment:
i modify this, add execption log. just reference formatUnixTimestamp，if
string args is not legal format,log and return null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] y0908105023 commented on a diff in pull request #20846: [FLINK-26943] [add date_add function support]
y0908105023 commented on code in PR #20846:
URL: https://github.com/apache/flink/pull/20846#discussion_r979893866
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/StringCallGen.scala:
##
@@ -190,6 +191,16 @@ object StringCallGen {
isCharacterString(operands(1).resultType) =>
methodGen(BuiltInMethods.STRING_TO_DATE_WITH_FORMAT)
+ case DATE_ADD
+ if operands.size == 2 &&
+!isCharacterString(operands.head.resultType) =>
+generateToDateAdd(ctx, operands, returnType)
+
+ case DATE_ADD
+ if operands.size == 2 &&
+isCharacterString(operands.head.resultType) =>
+generateToDateAdd(ctx, operands, returnType)
+
Review Comment:
To support string type and date type. eg: date_add(string type, offset) or
date_add(date type, offset)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-29131) Kubernetes operator webhook can use hostPort
[
https://issues.apache.org/jira/browse/FLINK-29131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609443#comment-17609443
]
Gyula Fora commented on FLINK-29131:
I agree changing the helm chart to have a single Deployment to 2 deployments
(operator + webhook separately) is a quite big change and will affect every
user who currently uses the webhook.
Before we go any further I think it might make sense to write a proposal on the
dev mailing list and discuss this openly + have a FLIP afterwards if there is a
general agreement.
> Kubernetes operator webhook can use hostPort
>
>
> Key: FLINK-29131
> URL: https://issues.apache.org/jira/browse/FLINK-29131
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.1.0
>Reporter: Dylan Meissner
>Assignee: Dylan Meissner
>Priority: Minor
>
> When running Flink operator on EKS cluster with Calico networking the
> control-plane (managed by AWS) cannot reach the webhook. Requests to create
> Flink resources fail with {_}Address is not allowed{_}.
> When the webhook listens on hostPort the requests to create Flink resources
> are successful. However, a pod security policy is generally required to allow
> webhook to listen on such ports.
> To support this scenario with the Helm chart make changes so that we can
> * Specify a hostPort value for the webhook
> * Name the port that the webhook listens on
> * Use the named port in the webhook service
> * Add a "use" pod security policy verb to cluster role
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-29417) One or more fetchers have encountered exception
[
https://issues.apache.org/jira/browse/FLINK-29417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-29417.
--
Resolution: Duplicate
> One or more fetchers have encountered exception
> ---
>
> Key: FLINK-29417
> URL: https://issues.apache.org/jira/browse/FLINK-29417
> Project: Flink
> Issue Type: Bug
>Reporter: liuchenhong
>Priority: Major
>
> One machine in the cluster goes offline, most tasks failover, and then it
> automatically restarts successfully. But the task of writing data to kudu
> failed to restart. The error is as follows:
>
> {code:java}
> //代码占位符
> 2022-09-26 16:06:30,634 WARN org.apache.flink.runtime.taskmanager.Task
> [] - KunlunRecLogOutputWindow (11/90)#6
> (9e23189bdc1ba38b19f922342819dab2) switched from RUNNING to FAILED with
> failure cause:
> org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
> Error at remote task manager '/***:**'.
> at
> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.decodeMsg(CreditBasedPartitionRequestClientHandler.java:351)
> at
> org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelRead(CreditBasedPartitionRequestClientHandler.java:240)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelRead(NettyMessageClientDecoderDelegate.java:112)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.flink.runtime.io.network.partition.ProducerFailedException:
> java.lang.RuntimeException: One or more fetchers have encountered exception
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.writeAndFlushNextMessageIfPossible(PartitionRequestQueue.java:285)
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.enqueueAvailableReader(PartitionRequestQueue.java:123)
> at
> org.apache.flink.runtime.io.network.netty.PartitionRequestQueue.userEventTriggered(PartitionRequestQueue.java:234)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117)
> at
>
[jira] [Commented] (FLINK-29199) Support blue-green deployment type
[
https://issues.apache.org/jira/browse/FLINK-29199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17609442#comment-17609442
]
Gyula Fora commented on FLINK-29199:
What I was saying is that cluster-id which must be unique within a namespace is
currently tied to the name of the FlinkDeployment.
If you create 2 flinkdeployments, they will have different names so different
cluster-ids. But since we cannot deploy 2 jobs using the same cluster-id for a
single FlinkDeployment we would have to completely rework this logic whihc
would have many implications to last-state recovery , HA etc.
> Support blue-green deployment type
> --
>
> Key: FLINK-29199
> URL: https://issues.apache.org/jira/browse/FLINK-29199
> Project: Flink
> Issue Type: New Feature
> Components: Kubernetes Operator
> Environment: Kubernetes
>Reporter: Oleg Vorobev
>Priority: Minor
>
> Are there any plans to support blue-green deployment/rollout mode similar to
> *BlueGreen* in the
> [flinkk8soperator|https://github.com/lyft/flinkk8soperator] to avoid downtime
> while updating?
> The idea is to run a new version in parallel with an old one and remove the
> old one only after the stability condition of the new one is satisfied (like
> in
> [rollbacks|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-release-1.1/docs/custom-resource/job-management/#application-upgrade-rollbacks-experimental]).
> For stateful apps with {*}upgradeMode: savepoint{*}, this means: not
> cancelling an old job after creating a savepoint -> starting new job from
> that savepoint -> waiting for it to become running/one successful
> checkpoint/timeout or something else -> cancelling and removing old job.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
