[GitHub] [flink] flinkbot edited a comment on pull request #17771: [FLINK-24813][table-planner]Improve ImplicitTypeConversionITCase
flinkbot edited a comment on pull request #17771: URL: https://github.com/apache/flink/pull/17771#issuecomment-966899848 ## CI report: * 7598282e780de754cad4a9b425ac6e7c008e4b9e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26582) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shenzhu commented on pull request #17788: [FLINK-15826][Tabel SQL/API] Add renameFunction() to Catalog
shenzhu commented on pull request #17788: URL: https://github.com/apache/flink/pull/17788#issuecomment-969977283 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] snuyanzin commented on a change in pull request #17786: [FLINK-24889][table] Flink SQL Client should print correсtly multisets
snuyanzin commented on a change in pull request #17786:
URL: https://github.com/apache/flink/pull/17786#discussion_r74741
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/MapAndMultisetToStringCastRule.java
##
@@ -109,16 +104,14 @@ protected String generateCodeBlockInternal(
String returnVariable,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
-boolean isMap = inputLogicalType instanceof MapType;
-final LogicalType keyType;
-final LogicalType valueType;
-if (isMap) {
-keyType = ((MapType) inputLogicalType).getKeyType();
-valueType = ((MapType) inputLogicalType).getValueType();
-} else {
-keyType = ((MultisetType) inputLogicalType).getElementType();
-valueType = INT().getLogicalType();
-}
+final LogicalType keyType =
+inputLogicalType.is(LogicalTypeFamily.COLLECTION)
Review comment:
makes sense, probably I didn't get your previous comment as you referred
to LogicalTypeFamily
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] shenzhu commented on pull request #17788: [FLINK-15826][Tabel SQL/API] Add renameFunction() to Catalog
shenzhu commented on pull request #17788: URL: https://github.com/apache/flink/pull/17788#issuecomment-969976164 > ## CI report: > * [b59b4a5](https://github.com/apache/flink/commit/b59b4a592c14990f7d916fa116d0a1cc8f5ed085) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26578) > > Bot commands Checked build logs to find error message ``` 2021-11-16T01:33:18.9201987Z Nov 16 01:33:18 [ERROR] Failed to execute goal on project flink-avro-confluent-registry: Could not resolve dependencies for project org.apache.flink:flink-avro-confluent-registry:jar:1.15-SNAPSHOT: Failed to collect dependencies at io.confluent:kafka-schema-registry-client:jar:5.5.2: Failed to read artifact descriptor for io.confluent:kafka-schema-registry-client:jar:5.5.2: Could not transfer artifact io.confluent:rest-utils-parent:pom:5.5.2 from/to confluent (https://packages.confluent.io/maven/): transfer failed for https://packages.confluent.io/maven/io/confluent/rest-utils-parent/5.5.2/rest-utils-parent-5.5.2.pom: Connection reset -> [Help 1] ``` Seems not related to changes in this PR, will trigger rerun. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24664) Support merge for JSON_ARRAYAGG
[ https://issues.apache.org/jira/browse/FLINK-24664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser updated FLINK-24664: --- Parent: (was: FLINK-9477) Issue Type: New Feature (was: Sub-task) > Support merge for JSON_ARRAYAGG > --- > > Key: FLINK-24664 > URL: https://issues.apache.org/jira/browse/FLINK-24664 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Reporter: Ingo Bürk >Priority: Major > > For JSON_ARRAYAGG we currently do not support merge (required e.g. for hop > windows) because arrays are order-sensitive, and we cannot ensure the correct > order. > There are a few ways we could solve this: > * Making use of WITHIN GROUP syntax, which is supported by Calcite, but not > yet for JSON_ARRAYAGG in the parser. > * Allowing incorrect behavior, but clearly documenting this case. In this > case we should at least try to find a way to have a deterministic order of > the accumulators to be merged as to not make the result non-deterministic. > * Try to actually solve this somehow such that behavior is correct. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24311) Support expression reduction for JSON construction functions
[
https://issues.apache.org/jira/browse/FLINK-24311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-24311:
---
Parent: (was: FLINK-9477)
Issue Type: New Feature (was: Sub-task)
> Support expression reduction for JSON construction functions
>
>
> Key: FLINK-24311
> URL: https://issues.apache.org/jira/browse/FLINK-24311
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API
>Reporter: Ingo Bürk
>Priority: Major
>
> For JSON construction functions such as JSON_OBJECT, we currently disable
> expression reduction. This is because these functions have special semantics
> where they behave differently depending on their call context. For example,
> JSON_OBJECT returns a JSON string. This means that a call like
> {code:java}
> JSON_OBJECT('A' VALUE JSON_OBJECT('B' VALUE 'C')){code}
> would result in
> {code:java}
> {"A": "{\"B\": \"C\"}"}{code}
> However, this is not user-friendly, and thus such nested calls are treated
> differently and instead result in the likely more intended outcome
> {code:java}
> {"A": {"B": "C"}}{code}
> To make this work, the function looks at its operands and checks whether each
> operand is another RexCall to such a JSON construction function. If it is, it
> inserts it as a raw node instead.
> This creates a problem during expression reduction. The RexCall will be
> replaced with a RexLiteral carrying the JSON string value. The function
> looking at the operands now cannot determine that this originated from such a
> RexCall anymore, and yields the unintended result once again. To prevent
> this, we currently disable expression reduction for these functions.
> We should aim to once again allow such expressions to be reduced while still
> preserving the intended behavior. See [this
> comment|https://github.com/apache/flink/pull/17186#issuecomment-920783089]
> for a rough idea of how this could be achieved.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Closed] (FLINK-24642) Implements unified iteration to support algorithms in flink-ml
[ https://issues.apache.org/jira/browse/FLINK-24642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao closed FLINK-24642. --- Resolution: Fixed > Implements unified iteration to support algorithms in flink-ml > -- > > Key: FLINK-24642 > URL: https://issues.apache.org/jira/browse/FLINK-24642 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Major > Fix For: 0.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24853) Report errors if jobs using iteration has runtime mode set to BATCH
[ https://issues.apache.org/jira/browse/FLINK-24853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-24853: Parent: (was: FLINK-24642) Issue Type: Improvement (was: Sub-task) > Report errors if jobs using iteration has runtime mode set to BATCH > > > Key: FLINK-24853 > URL: https://issues.apache.org/jira/browse/FLINK-24853 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Affects Versions: 0.1.0 >Reporter: Yun Gao >Assignee: Yun Gao >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-20188) Add Documentation for new File Source
[ https://issues.apache.org/jira/browse/FLINK-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-20188: -- Assignee: ZhuoYu Chen > Add Documentation for new File Source > - > > Key: FLINK-20188 > URL: https://issues.apache.org/jira/browse/FLINK-20188 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem, Documentation >Affects Versions: 1.14.0, 1.13.3, 1.15.0 >Reporter: Stephan Ewen >Assignee: ZhuoYu Chen >Priority: Blocker > Fix For: 1.15.0, 1.14.1, 1.13.4 > > Attachments: image-2021-11-16-11-42-32-957.png > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-20188) Add Documentation for new File Source
[ https://issues.apache.org/jira/browse/FLINK-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-20188: -- Assignee: (was: ZhuoYu Chen) > Add Documentation for new File Source > - > > Key: FLINK-20188 > URL: https://issues.apache.org/jira/browse/FLINK-20188 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem, Documentation >Affects Versions: 1.14.0, 1.13.3, 1.15.0 >Reporter: Stephan Ewen >Priority: Blocker > Fix For: 1.15.0, 1.14.1, 1.13.4 > > Attachments: image-2021-11-16-11-42-32-957.png > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-20188) Add Documentation for new File Source
[ https://issues.apache.org/jira/browse/FLINK-20188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444331#comment-17444331 ] Martijn Visser commented on FLINK-20188: [~monster#12] Yes, those files indeed contain the source code for the File Source. You can also check the JavaDocs of course at https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/connector/file/src/FileSource.html It would be great if you first could highlight how you would structure the documentation for the File Source. For example, the documentation for the File Sink has the following breakdown: * Introduction * File Formats ** Row-encoded formats ** Bulk-encoded Formats * Bucket Assignment * Rolling Policy * Part file lifecycle ** Part file configuration * Important Considerations ** General ** BATCH-specific ** S3-specific Do you have an idea how you would do this for the File Source? > Add Documentation for new File Source > - > > Key: FLINK-20188 > URL: https://issues.apache.org/jira/browse/FLINK-20188 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem, Documentation >Affects Versions: 1.14.0, 1.13.3, 1.15.0 >Reporter: Stephan Ewen >Priority: Blocker > Fix For: 1.15.0, 1.14.1, 1.13.4 > > Attachments: image-2021-11-16-11-42-32-957.png > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] slinkydeveloper commented on a change in pull request #17786: [FLINK-24889][table] Flink SQL Client should print correсtly multisets
slinkydeveloper commented on a change in pull request #17786:
URL: https://github.com/apache/flink/pull/17786#discussion_r749995285
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/MapAndMultisetToStringCastRule.java
##
@@ -109,16 +104,14 @@ protected String generateCodeBlockInternal(
String returnVariable,
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
-boolean isMap = inputLogicalType instanceof MapType;
-final LogicalType keyType;
-final LogicalType valueType;
-if (isMap) {
-keyType = ((MapType) inputLogicalType).getKeyType();
-valueType = ((MapType) inputLogicalType).getValueType();
-} else {
-keyType = ((MultisetType) inputLogicalType).getElementType();
-valueType = INT().getLogicalType();
-}
+final LogicalType keyType =
+inputLogicalType.is(LogicalTypeFamily.COLLECTION)
Review comment:
Shouldn't this be `inputLogicalType.is(LogicalTypeRoot.MULTISET)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-24810) Add Estimator and Model for the k-means clustering algorithm
[ https://issues.apache.org/jira/browse/FLINK-24810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao closed FLINK-24810. --- Fix Version/s: 0.1.0 Assignee: Dong Lin Resolution: Fixed > Add Estimator and Model for the k-means clustering algorithm > > > Key: FLINK-24810 > URL: https://issues.apache.org/jira/browse/FLINK-24810 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Dong Lin >Assignee: Dong Lin >Priority: Major > Labels: pull-request-available > Fix For: 0.1.0 > > > See [https://en.wikipedia.org/wiki/K-means_clustering] for the algorithm > description. > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24810) Add Estimator and Model for the k-means clustering algorithm
[ https://issues.apache.org/jira/browse/FLINK-24810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444329#comment-17444329 ] Yun Gao commented on FLINK-24810: - Fix on master via 212be99e5aaebabfc42ec74a9814dd1e8bde0b84 > Add Estimator and Model for the k-means clustering algorithm > > > Key: FLINK-24810 > URL: https://issues.apache.org/jira/browse/FLINK-24810 > Project: Flink > Issue Type: Improvement > Components: Library / Machine Learning >Reporter: Dong Lin >Priority: Major > Labels: pull-request-available > > See [https://en.wikipedia.org/wiki/K-means_clustering] for the algorithm > description. > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24802) Improve cast ROW to STRING
[ https://issues.apache.org/jira/browse/FLINK-24802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444328#comment-17444328 ] Francesco Guardiani commented on FLINK-24802: - That's all yours, thank you! If you need any help, feel free to ping me in the comments here. > Improve cast ROW to STRING > -- > > Key: FLINK-24802 > URL: https://issues.apache.org/jira/browse/FLINK-24802 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Assignee: Shen Zhu >Priority: Major > > When casting ROW to string, we should have a space after the comma to be > consistent with ARRAY, MAP, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-24918) Support to specify the data dir for state benchmark
[
https://issues.apache.org/jira/browse/FLINK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444327#comment-17444327
]
Aitozi commented on FLINK-24918:
Hi [~yunta] , do you have some inputs for this ?
> Support to specify the data dir for state benchmark
>
>
> Key: FLINK-24918
> URL: https://issues.apache.org/jira/browse/FLINK-24918
> Project: Flink
> Issue Type: Improvement
> Components: Benchmarks
>Reporter: Aitozi
>Priority: Minor
>
> {{StateBackendBenchmarkUtils}} use null as the parent dir to create temp dir,
> which will finally use the /tmp as the data dir. It has two downsides:
> 1. the /tmp dir often mount with tmpfs, which may store data in memory. it
> will affect the result of rocksdb benchmark
> 2. It can not support to use benchmark to measure the performance on a new
> device.
> So I purpose to enhance the state benchmark to support specify the default
> data dir. And avoiding to use the {{/tmp}} as default.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] gaoyunhaii closed pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
gaoyunhaii closed pull request #27: URL: https://github.com/apache/flink-ml/pull/27 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-24802) Improve cast ROW to STRING
[ https://issues.apache.org/jira/browse/FLINK-24802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francesco Guardiani reassigned FLINK-24802: --- Assignee: Shen Zhu > Improve cast ROW to STRING > -- > > Key: FLINK-24802 > URL: https://issues.apache.org/jira/browse/FLINK-24802 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Assignee: Shen Zhu >Priority: Major > > When casting ROW to string, we should have a space after the comma to be > consistent with ARRAY, MAP, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-24918) Support to specify the data dir for state benchmark
Aitozi created FLINK-24918:
--
Summary: Support to specify the data dir for state benchmark
Key: FLINK-24918
URL: https://issues.apache.org/jira/browse/FLINK-24918
Project: Flink
Issue Type: Improvement
Components: Benchmarks
Reporter: Aitozi
{{StateBackendBenchmarkUtils}} use null as the parent dir to create temp dir,
which will finally use the /tmp as the data dir. It has two downsides:
1. the /tmp dir often mount with tmpfs, which may store data in memory. it will
affect the result of rocksdb benchmark
2. It can not support to use benchmark to measure the performance on a new
device.
So I purpose to enhance the state benchmark to support specify the default data
dir. And avoiding to use the {{/tmp}} as default.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] lindong28 commented on pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
lindong28 commented on pull request #27: URL: https://github.com/apache/flink-ml/pull/27#issuecomment-969968812 Thanks for your review @gaoyunhaii @yunfengzhou-hub! All comments have been addressed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
lindong28 commented on a change in pull request #27: URL: https://github.com/apache/flink-ml/pull/27#discussion_r749986813 ## File path: flink-ml-lib/src/main/java/org/apache/flink/ml/param/HasFeaturesCol.java ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.param; Review comment: Discussed with @yunfengzhou-hub offline. We agreed there is no need to distinguish whether a param is a `colname` or not by having a dedicated `colname` subpackage. And we agreed to move those shared `WithParams` to the package `org.apache.flink.ml.common.param`. The `common` indicates `shared` just like Spark does. ## File path: flink-ml-lib/src/main/java/org/apache/flink/ml/param/HasFeaturesCol.java ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.param; Review comment: Discussed with @yunfengzhou-hub offline. We agreed there is no need to distinguish whether a param is a `colname` or not by having a dedicated `colname` subpackage. And we agreed to move those shared `WithParams` to the package `org.apache.flink.ml.common.param`. The word `common` indicates `shared`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17803: [FLINK-24839][fs-connector] Increase Timeout of FsStreamingSinkITCaseBase to 240
flinkbot edited a comment on pull request #17803: URL: https://github.com/apache/flink/pull/17803#issuecomment-969724257 ## CI report: * 3835826766a03449d004fe4b3893c89dbba05392 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26579) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17666: [FLINK-21327][table-planner-blink] Support window TVF in batch mode
flinkbot edited a comment on pull request #17666: URL: https://github.com/apache/flink/pull/17666#issuecomment-960514675 ## CI report: * 9420e4053db0f66aae31958cf0487a413016a5d3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26478) * 02f43393cff64c3635b683af8ea1318d53b24eab Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26587) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17806: [FLINK-24708][planner] Fix wrong results of the IN operator
flinkbot edited a comment on pull request #17806: URL: https://github.com/apache/flink/pull/17806#issuecomment-969962850 ## CI report: * 582c7a86a1deb13ec8fd0ac4223c21ca3c551fdd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26586) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #17666: [FLINK-21327][table-planner-blink] Support window TVF in batch mode
beyond1920 commented on a change in pull request #17666:
URL: https://github.com/apache/flink/pull/17666#discussion_r749980934
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/sql/SqlWindowTableFunction.java
##
@@ -275,16 +277,25 @@ boolean checkTableAndDescriptorOperands(SqlCallBinding
callBinding, int descript
SqlNameMatcher matcher =
validator.getCatalogReader().nameMatcher();
for (RelDataTypeField field : type.getFieldList()) {
if (matcher.matches(field.getName(), columnName)) {
-if (FlinkTypeFactory.isTimeIndicatorType(field.getType()))
{
+RelDataType fieldType = field.getType();
+if (FlinkTypeFactory.isTimeIndicatorType(fieldType)) {
return Optional.empty();
} else {
-ValidationException exception =
-new ValidationException(
-String.format(
-"The window function %s
requires the timecol is a time attribute type, but is %s.",
-
callBinding.getOperator().getAllowedSignatures(),
-field.getType()));
-return Optional.of(exception);
+LogicalType timeAttributeType =
FlinkTypeFactory.toLogicalType(fieldType);
+if (!canBeTimeAttributeType(timeAttributeType)) {
+ValidationException exception =
+new ValidationException(
+String.format(
+"The window function %s
requires the timecol is a time attribute type.\n"
++ "The supported
time indicator type are TIMESTAMP and TIMESTAMP_LTZ, but is %s.",
Review comment:
make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17666: [FLINK-21327][table-planner-blink] Support window TVF in batch mode
flinkbot edited a comment on pull request #17666: URL: https://github.com/apache/flink/pull/17666#issuecomment-960514675 ## CI report: * 9420e4053db0f66aae31958cf0487a413016a5d3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26478) * 02f43393cff64c3635b683af8ea1318d53b24eab UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17806: [FLINK-24708][planner] Fix wrong results of the IN operator
flinkbot commented on pull request #17806: URL: https://github.com/apache/flink/pull/17806#issuecomment-969963516 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 582c7a86a1deb13ec8fd0ac4223c21ca3c551fdd (Tue Nov 16 07:38:14 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17806: [FLINK-24708][planner] Fix wrong results of the IN operator
flinkbot commented on pull request #17806: URL: https://github.com/apache/flink/pull/17806#issuecomment-969962850 ## CI report: * 582c7a86a1deb13ec8fd0ac4223c21ca3c551fdd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 opened a new pull request #17806: [FLINK-24708][planner] Fix wrong results of the IN operator
fsk119 opened a new pull request #17806: URL: https://github.com/apache/flink/pull/17806 ## What is the purpose of the change *Fix the wrong results of the IN opreator. The reason behind is it uses the type of the first element in the SEARCH argument as the type of the other elements. It may change the value of the other elements. For example, it may rightPad the char element if the common type has larger size. Therefore, it recommends to use `RelDataTypeFactory` to calculate the least restrictive type for all types.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17771: [FLINK-24813][table-planner]Improve ImplicitTypeConversionITCase
flinkbot edited a comment on pull request #17771: URL: https://github.com/apache/flink/pull/17771#issuecomment-966899848 ## CI report: * 1412805a2c87c8eb9dd02e0bbc8f9c704e899730 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26580) * 7598282e780de754cad4a9b425ac6e7c008e4b9e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26582) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444320#comment-17444320 ] Arvid Heise commented on FLINK-21214: - On master, we bumped Kafka to 2.8.1 but we can't backport it to older release channels. I suggest to harden the test on old releases with retry rules. > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.13.3 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > Fix For: 1.15.0, 1.14.1 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (FLINK-18634) FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout expired after 60000milliseconds while awaiting InitProducerId"
[
https://issues.apache.org/jira/browse/FLINK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-18634:
---
Assignee: Fabian Paul (was: Arvid Heise)
> FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout
> expired after 6milliseconds while awaiting InitProducerId"
>
>
> Key: FLINK-18634
> URL: https://issues.apache.org/jira/browse/FLINK-18634
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Dian Fu
>Assignee: Fabian Paul
>Priority: Major
> Labels: auto-unassigned, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4590=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-07-17T11:43:47.9693015Z [ERROR] Tests run: 12, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 269.399 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> 2020-07-17T11:43:47.9693862Z [ERROR]
> testRecoverCommittedTransaction(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 60.679 s <<< ERROR!
> 2020-07-17T11:43:47.9694737Z org.apache.kafka.common.errors.TimeoutException:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> 2020-07-17T11:43:47.9695376Z Caused by:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-18634) FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout expired after 60000milliseconds while awaiting InitProducerId"
[
https://issues.apache.org/jira/browse/FLINK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444319#comment-17444319
]
Arvid Heise commented on FLINK-18634:
-
This seems to prevail even after we bumped Kafka to 2.8.1.
> FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout
> expired after 6milliseconds while awaiting InitProducerId"
>
>
> Key: FLINK-18634
> URL: https://issues.apache.org/jira/browse/FLINK-18634
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Dian Fu
>Assignee: Arvid Heise
>Priority: Major
> Labels: auto-unassigned, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4590=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-07-17T11:43:47.9693015Z [ERROR] Tests run: 12, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 269.399 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> 2020-07-17T11:43:47.9693862Z [ERROR]
> testRecoverCommittedTransaction(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 60.679 s <<< ERROR!
> 2020-07-17T11:43:47.9694737Z org.apache.kafka.common.errors.TimeoutException:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> 2020-07-17T11:43:47.9695376Z Caused by:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-20950) SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with "AssertionError"
[
https://issues.apache.org/jira/browse/FLINK-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444317#comment-17444317
]
Yun Gao commented on FLINK-20950:
-
Very thanks [~arvid] !
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with
> "AssertionError"
> ---
>
> Key: FLINK-20950
> URL: https://issues.apache.org/jira/browse/FLINK-20950
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.2, 1.13.0, 1.14.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11940=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2]
> {code:java}
> 2021-01-12T16:47:00.7579536Z [ERROR] Failures:
> 2021-01-12T16:47:00.7580061Z [ERROR]
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode:218
> 2021-01-12T16:47:00.7587186Z Expected: iterable over
> ["(895,null,-9223372036854775808)", "(895,null,-9223372036854775808)",
> "(127,null,-9223372036854775808)", "(127,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(161,null,-9223372036854775808)", "(161,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(662,null,-9223372036854775808)", "(662,null,-9223372036854775808)",
> "(822,null,-9223372036854775808)", "(822,null,-9223372036854775808)",
> "(491,null,-9223372036854775808)", "(491,null,-9223372036854775808)",
> "(275,null,-9223372036854775808)", "(275,null,-9223372036854775808)",
> "(122,null,-9223372036854775808)", "(122,null,-9223372036854775808)",
> "(850,null,-9223372036854775808)", "(850,null,-9223372036854775808)",
> "(630,null,-9223372036854775808)", "(630,null,-9223372036854775808)",
> "(682,null,-9223372036854775808)", "(682,null,-9223372036854775808)",
> "(765,null,-9223372036854775808)", "(765,null,-9223372036854775808)",
> "(434,null,-9223372036854775808)", "(434,null,-9223372036854775808)",
> "(970,null,-9223372036854775808)", "(970,null,-9223372036854775808)",
> "(714,null,-9223372036854775808)", "(714,null,-9223372036854775808)",
> "(795,null,-9223372036854775808)", "(795,null,-9223372036854775808)",
> "(288,null,-9223372036854775808)", "(288,null,-9223372036854775808)",
> "(422,null,-9223372036854775808)", "(422,null,-9223372036854775808)"] in any
> order
> 2021-01-12T16:47:00.7591663Z but: Not matched: "end of input"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17805: [FLINK-24708][planner] Fix wrong results of the IN operator
flinkbot edited a comment on pull request #17805: URL: https://github.com/apache/flink/pull/17805#issuecomment-969943622 ## CI report: * 067f050f59f186f274c40bc349d74aadd34a23bd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26585) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17788: [FLINK-15826][Tabel SQL/API] Add renameFunction() to Catalog
flinkbot edited a comment on pull request #17788: URL: https://github.com/apache/flink/pull/17788#issuecomment-968179643 ## CI report: * b59b4a592c14990f7d916fa116d0a1cc8f5ed085 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26578) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #17805: [FLINK-24708][planner] Fix wrong results of the IN operator
flinkbot commented on pull request #17805: URL: https://github.com/apache/flink/pull/17805#issuecomment-969945631 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 067f050f59f186f274c40bc349d74aadd34a23bd (Tue Nov 16 07:19:16 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20950) SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with "AssertionError"
[
https://issues.apache.org/jira/browse/FLINK-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444316#comment-17444316
]
Arvid Heise commented on FLINK-20950:
-
I can also take a look.
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with
> "AssertionError"
> ---
>
> Key: FLINK-20950
> URL: https://issues.apache.org/jira/browse/FLINK-20950
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.2, 1.13.0, 1.14.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11940=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2]
> {code:java}
> 2021-01-12T16:47:00.7579536Z [ERROR] Failures:
> 2021-01-12T16:47:00.7580061Z [ERROR]
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode:218
> 2021-01-12T16:47:00.7587186Z Expected: iterable over
> ["(895,null,-9223372036854775808)", "(895,null,-9223372036854775808)",
> "(127,null,-9223372036854775808)", "(127,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(161,null,-9223372036854775808)", "(161,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(662,null,-9223372036854775808)", "(662,null,-9223372036854775808)",
> "(822,null,-9223372036854775808)", "(822,null,-9223372036854775808)",
> "(491,null,-9223372036854775808)", "(491,null,-9223372036854775808)",
> "(275,null,-9223372036854775808)", "(275,null,-9223372036854775808)",
> "(122,null,-9223372036854775808)", "(122,null,-9223372036854775808)",
> "(850,null,-9223372036854775808)", "(850,null,-9223372036854775808)",
> "(630,null,-9223372036854775808)", "(630,null,-9223372036854775808)",
> "(682,null,-9223372036854775808)", "(682,null,-9223372036854775808)",
> "(765,null,-9223372036854775808)", "(765,null,-9223372036854775808)",
> "(434,null,-9223372036854775808)", "(434,null,-9223372036854775808)",
> "(970,null,-9223372036854775808)", "(970,null,-9223372036854775808)",
> "(714,null,-9223372036854775808)", "(714,null,-9223372036854775808)",
> "(795,null,-9223372036854775808)", "(795,null,-9223372036854775808)",
> "(288,null,-9223372036854775808)", "(288,null,-9223372036854775808)",
> "(422,null,-9223372036854775808)", "(422,null,-9223372036854775808)"] in any
> order
> 2021-01-12T16:47:00.7591663Z but: Not matched: "end of input"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-24917) YARNSessionFIFOSecuredITCase hangs on azure
Yun Gao created FLINK-24917:
---
Summary: YARNSessionFIFOSecuredITCase hangs on azure
Key: FLINK-24917
URL: https://issues.apache.org/jira/browse/FLINK-24917
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.14.0
Reporter: Yun Gao
{code:java}
Nov 15 23:50:37 [INFO] Running
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase
[Fatal Error] yarn-site.xml:1:1: Premature end of file.
Exception in thread "Thread-208" java.lang.RuntimeException:
org.xml.sax.SAXParseException; systemId:
file:/__w/1/s/flink-yarn-tests/target/test-classes/yarn-site.xml; lineNumber:
1; columnNumber: 1; Premature end of file.
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2338)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2185)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2102)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1068)
at
org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:139)
at
org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refresh(ServiceAuthorizationManager.java:126)
at org.apache.hadoop.ipc.Server.refreshServiceAcl(Server.java:454)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.createServer(ResourceLocalizationService.java:290)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:261)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:295)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:197)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper$1.run(MiniYARNCluster.java:561)
Caused by: org.xml.sax.SAXParseException; systemId:
file:/__w/1/s/flink-yarn-tests/target/test-classes/yarn-site.xml; lineNumber:
1; columnNumber: 1; Premature end of file.
at
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2173)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2161)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2232)
... 16 more
Nov 15, 2021 11:50:58 PM com.google.inject.servlet.GuiceFilter setPipeline
{code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26571=logs=f450c1a5-64b1-5955-e215-49cb1ad5ec88=cc452273-9efa-565d-9db8-ef62a38a0c10=29706
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #17805: [FLINK-24708][planner] Fix wrong results of the IN operator
flinkbot commented on pull request #17805: URL: https://github.com/apache/flink/pull/17805#issuecomment-969943622 ## CI report: * 067f050f59f186f274c40bc349d74aadd34a23bd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
yunfengzhou-hub commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749965528
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayes.java
##
@@ -0,0 +1,333 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.functions.ReduceFunction;
+import org.apache.flink.api.java.functions.KeySelector;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.tuple.Tuple3;
+import org.apache.flink.api.java.tuple.Tuple4;
+import org.apache.flink.ml.api.core.Estimator;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.functions.windowing.AllWindowFunction;
+import org.apache.flink.streaming.api.windowing.windows.TimeWindow;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+/**
+ * Naive Bayes classifier is a simple probability classification algorithm
using
+ * Bayes theorem based on independent assumption. It is an independent feature
model.
+ * The input feature can be continual or categorical.
+ */
+public class NaiveBayes implements Estimator,
+NaiveBayesParams {
+private final Map, Object> paramMap = new HashMap<>();
+
+@Override
+public NaiveBayesModel fit(Table... inputs) {
+String[] featureColNames = getFeatureCols();
+String labelColName = getLabelCol();
+String predictionCol = getPredictionCol();
+double smoothing = getSmoothing();
+
+Preconditions.checkNotNull(inputs, "input table list should not be
null");
+Preconditions.checkArgument(inputs.length == 1, "input table list
should contain only one argument");
+Preconditions.checkArgument(
+new HashSet<>(Arrays.asList(featureColNames)).size() ==
featureColNames.length,
Review comment:
I agree we can remove this check.
My consideration was that, like we want to throw NPE instead of ignoring
null values or labels, duplicated feature column also faces such choice. We
could explicitly throw an exception and let users handle it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17793: [FLINK-21565][Table SQL/API] Support more integer types in TIMESTAMPADD
flinkbot edited a comment on pull request #17793: URL: https://github.com/apache/flink/pull/17793#issuecomment-968529898 ## CI report: * 9c928c5f1a881c6851160bd209db056696ad619f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26577) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 opened a new pull request #17805: [FLINK-24708][planner] Fix wrong results of the IN operator
fsk119 opened a new pull request #17805: URL: https://github.com/apache/flink/pull/17805 ## What is the purpose of the change *Fix the wrong results of the IN opreator. The reason behind is it uses the type of the first element in the SEARCH argument as the type of the other elements. It may change the value of the other elements. For example, it may rightPad the char element if the common type has larger size. Therefore, it recommends to use `RelDataTypeFactory` to calculate the least restrictive type for all types.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24895) SqlTimestampSerializer#deserialize forgets to fully read from source when the record is null.
[ https://issues.apache.org/jira/browse/FLINK-24895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444312#comment-17444312 ] Timo Walther commented on FLINK-24895: -- Thanks, we will find someone who can fix this. > SqlTimestampSerializer#deserialize forgets to fully read from source when the > record is null. > - > > Key: FLINK-24895 > URL: https://issues.apache.org/jira/browse/FLINK-24895 > Project: Flink > Issue Type: Bug > Components: API / Type Serialization System >Reporter: bx123 >Priority: Minor > > As we support null Timestamp in this Serializer. In the 2 deserialize() > methods, when we read the first long and get Long.MIN_VALUE, we directly > return null and forget to read the next int which responds to nanos. > If there is really a null Timestamp that has been serialized before, we may > get wrong value when we deserialize the Object after this timestamp. So > before return null, source.skipBytesToRead(4) is needed to make the source > in the right position. > By the way, I find the copy(source, target) is implemented by deserialize the > intermediate value, while its more efficient to just copy 12 bytes from > source to target like target.write(source, 12). > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] beyond1920 commented on a change in pull request #17733: [FLINK-24835][table-planner] Fix bug in `RelTimeIndicatorConverter` when materialize time attribute fields of regular join's i
beyond1920 commented on a change in pull request #17733:
URL: https://github.com/apache/flink/pull/17733#discussion_r749961893
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/WindowJoinUtil.scala
##
@@ -60,7 +61,11 @@ object WindowJoinUtil extends Logging {
* ends equality of input tables, else false.
*/
def satisfyWindowJoin(join: FlinkLogicalJoin): Boolean = {
-excludeWindowStartEqualityAndEndEqualityFromJoinInfoPairs(join) match {
+satisfyWindowJoin(join, join.getLeft, join.getRight)
Review comment:
ditto
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/TemporalJoinUtil.scala
##
@@ -424,10 +425,14 @@ object TemporalJoinUtil {
* else false.
*/
def satisfyTemporalJoin(join: FlinkLogicalJoin): Boolean = {
+satisfyTemporalJoin(join, join.getLeft, join.getRight)
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #17733: [FLINK-24835][table-planner] Fix bug in `RelTimeIndicatorConverter` when materialize time attribute fields of regular join's i
beyond1920 commented on a change in pull request #17733:
URL: https://github.com/apache/flink/pull/17733#discussion_r749961611
##
File path:
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/IntervalJoinUtil.scala
##
@@ -450,15 +452,26 @@ object IntervalJoinUtil {
* else false.
*/
def satisfyIntervalJoin(join: FlinkLogicalJoin): Boolean = {
+satisfyIntervalJoin(join, join.getLeft, join.getRight)
Review comment:
If no new left node and right node, there is no need to pass the
original left and right again because we could get left/right from join node.
It's a shortcut, besides, does not introduce unnessassary change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-24764) pyflink/table/tests/test_udf.py hang on Azure
[
https://issues.apache.org/jira/browse/FLINK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444309#comment-17444309
]
Yun Gao commented on FLINK-24764:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26571=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23620]
> pyflink/table/tests/test_udf.py hang on Azure
> -
>
> Key: FLINK-24764
> URL: https://issues.apache.org/jira/browse/FLINK-24764
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.5, 1.15.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1, 1.13.4
>
>
> {{pyflink/table/tests/test_udf.py}} seems to hang on Azure.
> {code}
> 2021-11-04T03:12:01.4537829Z py36-cython run-test: commands[3] | pytest
> --durations=20
> 2021-11-04T03:12:03.6955063Z = test session
> starts ==
> 2021-11-04T03:12:03.6957461Z platform linux -- Python 3.6.13, pytest-6.2.5,
> py-1.10.0, pluggy-1.0.0
> 2021-11-04T03:12:03.6959285Z cachedir: .tox/py36-cython/.pytest_cache
> 2021-11-04T03:12:03.6960653Z rootdir: /__w/1/s/flink-python
> 2021-11-04T03:12:03.6961356Z collected 690 items
> 2021-11-04T03:12:03.6961755Z
> 2021-11-04T03:12:04.6615796Z pyflink/common/tests/test_configuration.py
> ..[ 1%]
> 2021-11-04T03:12:04.9315499Z pyflink/common/tests/test_execution_config.py
> ...[ 4%]
> 2021-11-04T03:12:05.4226061Z
> pyflink/common/tests/test_serialization_schemas.py ... [
> 5%]
> 2021-11-04T03:12:05.8920762Z pyflink/common/tests/test_typeinfo.py ...
> [ 5%]
> 2021-11-04T03:12:10.3843622Z
> pyflink/dataset/tests/test_execution_environment.py ...s.[
> 6%]
> 2021-11-04T03:12:10.4385641Z
> pyflink/dataset/tests/test_execution_environment_completeness.py . [
> 7%]
> 2021-11-04T03:12:10.5390180Z
> pyflink/datastream/tests/test_check_point_config.py ... [
> 8%]
> 2021-11-04T03:12:20.1148835Z pyflink/datastream/tests/test_connectors.py ...
> [ 9%]
> 2021-11-04T03:13:12.4436977Z pyflink/datastream/tests/test_data_stream.py
> ... [ 13%]
> 2021-11-04T03:13:22.6815256Z
> [ 14%]
> 2021-11-04T03:13:22.9777981Z pyflink/datastream/tests/test_state_backend.py
> ..[ 16%]
> 2021-11-04T03:13:33.4281095Z
> pyflink/datastream/tests/test_stream_execution_environment.py .. [
> 18%]
> 2021-11-04T03:13:45.3707210Z .s.
> [ 21%]
> 2021-11-04T03:13:45.5100419Z
> pyflink/datastream/tests/test_stream_execution_environment_completeness.py .
> [ 21%]
> 2021-11-04T03:13:45.5107357Z
> [ 21%]
> 2021-11-04T03:13:45.5824541Z pyflink/fn_execution/tests/test_coders.py
> s [ 24%]
> 2021-11-04T03:13:45.6311670Z pyflink/fn_execution/tests/test_fast_coders.py
> ... [ 27%]
> 2021-11-04T03:13:45.6480686Z
> pyflink/fn_execution/tests/test_flink_fn_execution_pb2_synced.py . [
> 27%]
> 2021-11-04T03:13:48.3033527Z
> pyflink/fn_execution/tests/test_process_mode_boot.py ... [
> 28%]
> 2021-11-04T03:13:48.3169538Z pyflink/metrics/tests/test_metric.py .
> [ 28%]
> 2021-11-04T03:13:48.3928810Z pyflink/ml/tests/test_ml_environment.py ...
> [ 29%]
> 2021-11-04T03:13:48.4381082Z pyflink/ml/tests/test_ml_environment_factory.py
> ... [ 29%]
> 2021-11-04T03:13:48.4696143Z pyflink/ml/tests/test_params.py .
> [ 31%]
> 2021-11-04T03:13:48.5140301Z pyflink/ml/tests/test_pipeline.py
> [ 32%]
> 2021-11-04T03:13:50.2573824Z pyflink/ml/tests/test_pipeline_it_case.py ...
> [ 32%]
> 2021-11-04T03:13:50.3598135Z pyflink/ml/tests/test_pipeline_stage.py ..
> [ 32%]
> 2021-11-04T03:14:18.5397420Z pyflink/table/tests/test_aggregate.py .
> [ 34%]
> 2021-11-04T03:14:20.1852937Z pyflink/table/tests/test_calc.py ...
> [ 35%]
> 2021-11-04T03:14:21.3674525Z pyflink/table/tests/test_catalog.py
> [ 40%]
> 2021-11-04T03:14:22.4375814Z ...
> [ 46%]
> 2021-11-04T03:14:22.4966492Z
[jira] [Assigned] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Xingbo reassigned FLINK-24495:
Assignee: Huang Xingbo
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444307#comment-17444307 ] Yun Gao commented on FLINK-21214: - [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26570=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5=6580] > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.13.3 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > Fix For: 1.15.0, 1.14.1 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-21214) FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed
[ https://issues.apache.org/jira/browse/FLINK-21214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21214: Affects Version/s: 1.13.3 > FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint Failed > -- > > Key: FLINK-21214 > URL: https://issues.apache.org/jira/browse/FLINK-21214 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.13.3 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > Fix For: 1.15.0, 1.14.1 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12687=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5] > > [ERROR] > testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase) > Time elapsed: 62.857 s <<< ERROR! > org.apache.kafka.common.errors.TimeoutException: > org.apache.kafka.common.errors.TimeoutException: Timeout expired after > 6milliseconds while awaiting InitProducerId > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout expired > after 6milliseconds while awaiting InitProducerId > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-22342) FlinkKafkaProducerITCase fails with producer leak
[
https://issues.apache.org/jira/browse/FLINK-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-22342:
Affects Version/s: 1.13.3
> FlinkKafkaProducerITCase fails with producer leak
> -
>
> Key: FLINK-22342
> URL: https://issues.apache.org/jira/browse/FLINK-22342
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.11.3, 1.14.0, 1.13.3
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16732=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20=6386
> {code}
> [ERROR]
> testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 8.854 s <<< FAILURE!
> java.lang.AssertionError: Detected producer leak. Thread name:
> kafka-producer-network-thread |
> producer-MockTask-002a002c-11
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.checkProducerLeak(FlinkKafkaProducerITCase.java:728)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint(FlinkKafkaProducerITCase.java:381)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-22342) FlinkKafkaProducerITCase fails with producer leak
[
https://issues.apache.org/jira/browse/FLINK-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-22342:
Affects Version/s: (was: 1.11.3)
> FlinkKafkaProducerITCase fails with producer leak
> -
>
> Key: FLINK-22342
> URL: https://issues.apache.org/jira/browse/FLINK-22342
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.14.0, 1.13.3
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16732=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20=6386
> {code}
> [ERROR]
> testScaleDownBeforeFirstCheckpoint(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 8.854 s <<< FAILURE!
> java.lang.AssertionError: Detected producer leak. Thread name:
> kafka-producer-network-thread |
> producer-MockTask-002a002c-11
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.checkProducerLeak(FlinkKafkaProducerITCase.java:728)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase.testScaleDownBeforeFirstCheckpoint(FlinkKafkaProducerITCase.java:381)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17613: [FLINK-24536][Table SQL/Planner] flink sql support bang equal '!='
flinkbot edited a comment on pull request #17613: URL: https://github.com/apache/flink/pull/17613#issuecomment-955719048 ## CI report: * d2fb15cc1d92c0ecd9428869d6e94bbbea9f4a04 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26421) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
yunfengzhou-hub commented on a change in pull request #32: URL: https://github.com/apache/flink-ml/pull/32#discussion_r749958347 ## File path: flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java ## @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.classification.naivebayes; + +import org.apache.flink.ml.param.DoubleParam; +import org.apache.flink.ml.param.Param; +import org.apache.flink.ml.param.ParamValidators; +import org.apache.flink.ml.param.shared.colname.HasFeatureCols; +import org.apache.flink.ml.param.shared.colname.HasLabelCol; +import org.apache.flink.ml.param.shared.colname.HasPredictionCol; + +/** + * Parameters of naive bayes training process. + */ +public interface NaiveBayesParams extends Review comment: OK. I'll add this param to NaiveBayes. One consideration is that while spark supports setting the type of a whole Naive Bayes model, Alink allows finer granularity to define the model type for each feature column. I agree that we just need to define the type of a whole model in the foreseeable future, but if users need to define type for each column they will need to create multiple NaiveBayesModel and use some bagging algorithm to unify them into one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24789) IllegalStateException with CheckpointCleaner being closed already
[
https://issues.apache.org/jira/browse/FLINK-24789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444304#comment-17444304
]
Yun Gao commented on FLINK-24789:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26573=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=7820]
> IllegalStateException with CheckpointCleaner being closed already
> -
>
> Key: FLINK-24789
> URL: https://issues.apache.org/jira/browse/FLINK-24789
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.14.0
>Reporter: Matthias
>Assignee: David Morávek
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
> Attachments:
> logs-ci_build-test_ci_build_finegrained_resource_management-1635785399.zip
>
>
> * We experienced a failure of {{OperatorCoordinatorSchedulerTest}} in our VVP
> Fork of Flink. The {{finegrained_resource_management}} test run failed with
> an non-0 exit code:
> {code}
> Nov 01 17:19:12 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.2:test (default-test) on
> project flink-runtime: There are test failures.
> Nov 01 17:19:12 [ERROR]
> Nov 01 17:19:12 [ERROR] Please refer to
> /__w/1/s/flink-runtime/target/surefire-reports for the individual test
> results.
> Nov 01 17:19:12 [ERROR] Please refer to dump files (if any exist)
> [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Nov 01 17:19:12 [ERROR] ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> Nov 01 17:19:12 [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/1/s/flink-runtime/target/surefire/surefirebooter6007815607334336440.jar
> /__w/1/s/flink-runtime/target/surefire 2021-11-01T16-51-51_363-jvmRun2
> surefire6448660128033443499tmp surefire_4131168043975619749001tmp
> Nov 01 17:19:12 [ERROR] Error occurred in starting fork, check output in log
> Nov 01 17:19:12 [ERROR] Process Exit Code: 239
> Nov 01 17:19:12 [ERROR] Crashed tests:
> Nov 01 17:19:12 [ERROR]
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorSchedulerTest
> Nov 01 17:19:12 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Nov 01 17:19:12 [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -jar
> /__w/1/s/flink-runtime/target/surefire/surefirebooter6007815607334336440.jar
> /__w/1/s/flink-runtime/target/surefire 2021-11-01T16-51-51_363-jvmRun2
> surefire6448660128033443499tmp surefire_4131168043975619749001tmp
> Nov 01 17:19:12 [ERROR] Error occurred in starting fork, check output in log
> Nov 01 17:19:12 [ERROR] Process Exit Code: 239
> Nov 01 17:19:12 [ERROR] Crashed tests:
> Nov 01 17:19:12 [ERROR]
> org.apache.flink.runtime.operators.coordination.OperatorCoordinatorSchedulerTest
> Nov 01 17:19:12 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> Nov 01 17:19:12 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
> {code}
> It looks like the {{testSnapshotAsyncFailureFailsCheckpoint}} caused it even
> though finishing successfully due to a fatal error when shutting down the
> cluster:
> {code}
> 17:07:27,264 [Checkpoint Timer] ERROR
> org.apache.flink.util.FatalExitExceptionHandler [] - FATAL:
> Thread 'Checkpoint Timer' produced an uncaught exception. Stopping the
> process...
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException: java.lang.IllegalStateException:
> CheckpointsCleaner has already been closed
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$startTriggeringCheckpoint$7(CheckpointCoordinator.java:626)
> ~[classes/:?]
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> ~[?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> [?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
> [?:1.8.0_292]
> at
>
[GitHub] [flink] XComp commented on a change in pull request #17781: [FLINK-24875][ci] Split E2E job
XComp commented on a change in pull request #17781:
URL: https://github.com/apache/flink/pull/17781#discussion_r749950868
##
File path: tools/azure-pipelines/e2e-template.yml
##
@@ -0,0 +1,131 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+parameters:
+ e2e_pool_definion: # defines the hardware pool for end-to-end test execution
+ stage_name: # defines a unique identifier for all jobs in a stage (in case
the jobs are added multiple times to a stage)
+ environment: # defines environment variables for downstream scripts
+ run_end_to_end: # if set to 'true', the end to end tests will be executed
+ jdk: # the jdk version to use
+ group: # the group of tests that should run
+
+jobs:
+- job: e2e_${{parameters.group}}_${{parameters.stage_name}}
+ # uncomment below condition to run the e2e tests only on request.
+ #condition: or(eq(variables['MODE'], 'e2e'),
eq(${{parameters.run_end_to_end}}, 'true'))
+ # We are running this in a separate pool
+ pool: ${{parameters.e2e_pool_definition}}
+ timeoutInMinutes: 310
+ cancelTimeoutInMinutes: 1
+ workspace:
+clean: all
+ steps:
+# Skip e2e test execution if this is a documentation only pull request
(master / release builds will still be checked regularly)
+- bash: |
+source ./tools/azure-pipelines/build_properties.sh
+is_docs_only_pullrequest
+if [[ "$?" == 0 ]] ; then
+ echo "##[debug]This is a documentation-only change. Skipping e2e
execution."
+ echo "##vso[task.setvariable variable=skip;]1"
+else
+ echo "##[debug]This is a regular CI build. Continuing ..."
+ echo "##vso[task.setvariable variable=skip;]0"
+fi
+ displayName: Check if Docs only PR
+# the cache task does not create directories on a cache miss, and can
later fail when trying to tar the directory if the test haven't created it
+# this may for example happen if a given directory is only used by a
subset of tests, which are run in a different 'group'
+- bash: |
+mkdir -p $(MAVEN_CACHE_FOLDER)
Review comment:
Why didn't we need these in the previous version in `jobs-template.yml`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17613: [FLINK-24536][Table SQL/Planner] flink sql support bang equal '!='
flinkbot edited a comment on pull request #17613: URL: https://github.com/apache/flink/pull/17613#issuecomment-955719048 ## CI report: * d2fb15cc1d92c0ecd9428869d6e94bbbea9f4a04 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26421) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
lindong28 commented on a change in pull request #27:
URL: https://github.com/apache/flink-ml/pull/27#discussion_r749956898
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/clustering/kmeans/KMeansModel.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.clustering.kmeans;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.distance.DistanceMeasure;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** A Model which clusters data into k clusters using the model data computed
by {@link KMeans}. */
+public class KMeansModel implements Model,
KMeansModelParams {
+private final Map, Object> paramMap = new HashMap<>();
+private Table centroidsTable;
+
+public KMeansModel() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public KMeansModel setModelData(Table... inputs) {
+centroidsTable = inputs[0];
+return this;
+}
+
+@Override
+public Table[] getModelData() {
+return new Table[] {centroidsTable};
+}
+
+@Override
+public Table[] transform(Table... inputs) {
+StreamTableEnvironment tEnv =
+(StreamTableEnvironment) ((TableImpl)
inputs[0]).getTableEnvironment();
+DataStream centroids =
+tEnv.toDataStream(centroidsTable).map(row -> (DenseVector[])
row.getField("f0"));
+
+String featureCol = getFeaturesCol();
+String predictionCol = getPredictionCol();
+DistanceMeasure distanceMeasure =
DistanceMeasure.getInstance(getDistanceMeasure());
+
+RowTypeInfo inputTypeInfo =
TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+RowTypeInfo outputTypeInfo =
+new RowTypeInfo(
+ArrayUtils.addAll(inputTypeInfo.getFieldTypes(),
Types.INT),
+ArrayUtils.addAll(inputTypeInfo.getFieldNames(),
predictionCol));
+
+DataStream input = tEnv.toDataStream(inputs[0]);
+DataStream output =
+input.connect(centroids.broadcast())
+.transform(
+"SelectNearestCentroid",
+outputTypeInfo,
+
[jira] [Commented] (FLINK-24801) "Post-job: Cache Maven local repo" failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-24801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444302#comment-17444302
]
Yun Gao commented on FLINK-24801:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26573=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=fef5355a-19b0-45cd-b087-ced61003e0e5=217]
> "Post-job: Cache Maven local repo" failed on Azure
> --
>
> Key: FLINK-24801
> URL: https://issues.apache.org/jira/browse/FLINK-24801
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> {code:java}
> 2021-11-05T13:49:20.5298458Z Resolved to:
> maven|Linux|kL7EJ8TeMrJ0VZs51DUWRqheXcKK2cN2spGtx9IbVxQ=
> 2021-11-05T13:49:21.0445785Z ApplicationInsightsTelemetrySender will
> correlate events with X-TFS-Session b5cbe6a0-61e7-4345-81b6-efaed3924cbe
> 2021-11-05T13:49:21.0700758Z Getting a pipeline cache artifact with one of
> the following fingerprints:
> 2021-11-05T13:49:21.0702157Z Fingerprint:
> `maven|Linux|kL7EJ8TeMrJ0VZs51DUWRqheXcKK2cN2spGtx9IbVxQ=`
> 2021-11-05T13:49:21.3648278Z There is a cache miss.
> 2021-11-05T13:50:26.4782603Z tar:
> c9692460fbd54c808fca7be315d83578_archive.tar: Wrote only 2048 of 10240 bytes
> 2021-11-05T13:50:26.4784975Z tar: Error is not recoverable: exiting now
> 2021-11-05T13:50:27.0397318Z ApplicationInsightsTelemetrySender correlated 1
> events with X-TFS-Session b5cbe6a0-61e7-4345-81b6-efaed3924cbe
> 2021-11-05T13:50:27.0531774Z ##[error]Process returned non-zero exit code: 2
> 2021-11-05T13:50:27.0666804Z ##[section]Finishing: Cache Maven local repo
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26018=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=96bd9872-da2e-43b4-b013-1295f1c23a41=220]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] lindong28 commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
lindong28 commented on a change in pull request #27:
URL: https://github.com/apache/flink-ml/pull/27#discussion_r749956301
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/MapPartitionFunctionWrapper.java
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Collector;
+
+/**
+ * This utility class wraps a MapPartitionFunction into an
OneInputStreamOperator so that a
+ * MapPartitionFunction can be applied on a DataStream via the
DataStream::transform API.
+ *
+ * @param The class type of the input element.
+ * @param The class type of the output element.
+ */
+public class MapPartitionFunctionWrapper extends
AbstractStreamOperator
+implements OneInputStreamOperator, BoundedOneInput {
+private final ListStateDescriptor descriptor;
+private final MapPartitionFunction mapPartitionFunc;
+private ListState values;
+
+public MapPartitionFunctionWrapper(
+TypeInformation typeInfo, MapPartitionFunction
mapPartitionFunc) {
+this.descriptor = new ListStateDescriptor<>("input", typeInfo);
Review comment:
Nice!
It is updated as suggestd.
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/common/datastream/MapPartitionFunctionWrapper.java
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.common.datastream;
+
+import org.apache.flink.api.common.functions.MapPartitionFunction;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.BoundedOneInput;
+import org.apache.flink.streaming.api.operators.OneInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.util.Collector;
+
+/**
+ * This utility class wraps a MapPartitionFunction into an
OneInputStreamOperator so that a
+ * MapPartitionFunction can be applied on a DataStream via the
DataStream::transform API.
+ *
+ * @param The class type of the input element.
+ * @param The class type of the output element.
+ */
+public class MapPartitionFunctionWrapper extends
AbstractStreamOperator
+implements OneInputStreamOperator, BoundedOneInput {
+private final ListStateDescriptor descriptor;
+private final MapPartitionFunction mapPartitionFunc;
+private ListState values;
+
+public MapPartitionFunctionWrapper(
+TypeInformation typeInfo, MapPartitionFunction
mapPartitionFunc) {
+
[GitHub] [flink-ml] lindong28 commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
lindong28 commented on a change in pull request #27:
URL: https://github.com/apache/flink-ml/pull/27#discussion_r749956101
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/param/HasFeaturesCol.java
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.param;
+
+/** Interface for the shared featuresCol param. */
+public interface HasFeaturesCol extends WithParams {
+Param FEATURES_COL = new StringParam("featuresCol", "Features
column name", "features");
+
+default String getFeaturesCol() {
Review comment:
It is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] beyond1920 commented on a change in pull request #17734: [FLINK-23305][table]Support semi/anti interval joins
beyond1920 commented on a change in pull request #17734:
URL: https://github.com/apache/flink/pull/17734#discussion_r749956116
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/join/interval/SemiAntiTimeIntervalJoin.java
##
@@ -0,0 +1,450 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.join.interval;
+
+import org.apache.flink.api.common.functions.RuntimeContext;
+import org.apache.flink.api.common.state.MapState;
+import org.apache.flink.api.common.state.MapStateDescriptor;
+import org.apache.flink.api.common.state.StateTtlConfig;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.api.java.typeutils.MapTypeInfo;
+import org.apache.flink.api.java.typeutils.TupleTypeInfo;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.functions.co.KeyedCoProcessFunction;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.generated.JoinCondition;
+import org.apache.flink.table.runtime.typeutils.InternalTypeInfo;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Collector;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static
org.apache.flink.table.runtime.util.StateConfigUtil.createTtlConfig;
+
+/**
+ * A CoProcessFunction to execute time interval (time-bounded) stream
semi/anti-join. Two kinds of
+ * time criteria: "L.time between R.time + X and R.time + Y" or "R.time
between L.time - Y and
+ * L.time - X" X and Y might be negative or positive and X <= Y.
+ */
+public abstract class SemiAntiTimeIntervalJoin
+extends KeyedCoProcessFunction {
+
+protected static final String LEFT_RECORDS_STATE_NAME = "left-records";
+protected static final String RIGHT_RECORDS_STATE_NAME = "right-records";
+
+private final boolean isAntiJoin;
+
+protected final long leftRelativeSize;
+protected final long rightRelativeSize;
+
+protected final long allowedLateness;
+private final InternalTypeInfo leftType;
+private final InternalTypeInfo rightType;
+private final IntervalJoinFunction joinFunction;
+private final long stateRetentionTime;
+
+// state to store rows from the left stream
+private transient SemiAntiLeftRecordStateView leftRecordStateView;
+// state to store rows from the right stream
+private transient SemiAntiRightRecordStateView rightRecordStateView;
+
+// Current time on the respective input stream.
+protected long leftOperatorTime = 0L;
+protected long rightOperatorTime = 0L;
+
+public SemiAntiTimeIntervalJoin(
+boolean isAntiJoin,
+long leftLowerBound,
+long leftUpperBound,
+long allowedLateness,
+InternalTypeInfo leftType,
+InternalTypeInfo rightType,
+IntervalJoinFunction joinFunction,
+long stateRetentionTime) {
+this.isAntiJoin = isAntiJoin;
+this.leftRelativeSize = -leftLowerBound;
+this.rightRelativeSize = leftUpperBound;
+if (allowedLateness < 0) {
+throw new IllegalArgumentException("The allowed lateness must be
non-negative.");
+}
+this.allowedLateness = allowedLateness;
+this.leftType = leftType;
+this.rightType = rightType;
+this.joinFunction = joinFunction;
+this.stateRetentionTime = stateRetentionTime;
+}
+
+@Override
+public void open(Configuration parameters) throws Exception {
+super.open(parameters);
+joinFunction.setRuntimeContext(getRuntimeContext());
+joinFunction.open(parameters);
+
+// TODO: support joins with unique keys
+// now states only support no unique key
+this.leftRecordStateView =
+new SemiAntiLeftRecordStateView(
+getRuntimeContext(),
+LEFT_RECORDS_STATE_NAME,
+leftType,
+createTtlConfig(stateRetentionTime));
+
+
[GitHub] [flink-ml] lindong28 commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
lindong28 commented on a change in pull request #27:
URL: https://github.com/apache/flink-ml/pull/27#discussion_r749956004
##
File path:
flink-ml-api/src/main/java/org/apache/flink/ml/linalg/typeinfo/DenseVectorSerializer.java
##
@@ -0,0 +1,133 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.flink.ml.linalg.typeinfo;
+
+import org.apache.flink.api.common.typeutils.SimpleTypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.TypeSerializerSnapshot;
+import org.apache.flink.api.common.typeutils.base.TypeSerializerSingleton;
+import org.apache.flink.core.memory.DataInputView;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.ml.linalg.DenseVector;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+/** Specialized serializer for {@code DenseVector}. */
+public final class DenseVectorSerializer extends
TypeSerializerSingleton {
+
+private static final long serialVersionUID = 1L;
+
+private static final double[] EMPTY = new double[0];
+
+private static final DenseVectorSerializer INSTANCE = new
DenseVectorSerializer();
+
+@Override
+public boolean isImmutableType() {
+return false;
+}
+
+@Override
+public DenseVector createInstance() {
+return new DenseVector(EMPTY);
+}
+
+@Override
+public DenseVector copy(DenseVector from) {
+return new DenseVector(Arrays.copyOf(from.values, from.values.length));
+}
+
+@Override
+public DenseVector copy(DenseVector from, DenseVector reuse) {
+if (from.values.length == reuse.values.length) {
+System.arraycopy(from.values, 0, reuse.values, 0,
from.values.length);
+return reuse;
+}
+return copy(from);
+}
+
+@Override
+public int getLength() {
+return -1;
+}
+
+@Override
+public void serialize(DenseVector vector, DataOutputView target) throws
IOException {
+if (vector == null) {
+throw new IllegalArgumentException("The vector must not be null.");
+}
+
+final int len = vector.values.length;
+target.writeInt(len);
+for (int i = 0; i < len; i++) {
+target.writeDouble(vector.get(i));
+}
+}
+
+@Override
+public DenseVector deserialize(DataInputView source) throws IOException {
+int len = source.readInt();
+double[] values = new double[len];
+for (int i = 0; i < len; i++) {
+values[i] = source.readDouble();
+}
+return new DenseVector(values);
+}
+
+@Override
+public DenseVector deserialize(DenseVector reuse, DataInputView source)
throws IOException {
+int len = source.readInt();
+if (len == reuse.values.length) {
+for (int i = 0; i < len; i++) {
+reuse.values[i] = source.readDouble();
+}
+return reuse;
+}
+
+double[] values = new double[len];
Review comment:
I agree. It is updated as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
lindong28 commented on a change in pull request #27:
URL: https://github.com/apache/flink-ml/pull/27#discussion_r749955817
##
File path:
flink-ml-api/src/main/java/org/apache/flink/ml/distance/DistanceMeasure.java
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.distance;
+
+import org.apache.flink.ml.linalg.Vector;
+
+import java.io.Serializable;
+
+/** Interface for measuring distance between two vectors. */
+public interface DistanceMeasure extends Serializable {
+
+static DistanceMeasure getInstance(String distanceMeasure) {
+if (distanceMeasure.equals("euclidean")) {
+return EuclideanDistanceMeasure.getInstance();
+}
+throw new IllegalArgumentException(
Review comment:
Good point. It is updated as suggested.
##
File path: flink-ml-api/src/main/java/org/apache/flink/ml/linalg/Vectors.java
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.linalg;
+
+/** Utility methods for instantiating Vector. */
+public class Vectors {
+
+/** Creates a dense vector from its values. */
+public static DenseVector dense(double... value) {
Review comment:
Thanks. It is fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-18634) FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout expired after 60000milliseconds while awaiting InitProducerId"
[
https://issues.apache.org/jira/browse/FLINK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444300#comment-17444300
]
Yun Gao commented on FLINK-18634:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=1fc6e7bf-633c-5081-c32a-9dea24b05730=576aba0a-d787-51b6-6a92-cf233f360582=7153]
> FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout
> expired after 6milliseconds while awaiting InitProducerId"
>
>
> Key: FLINK-18634
> URL: https://issues.apache.org/jira/browse/FLINK-18634
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0, 1.13.0, 1.14.0, 1.15.0
>Reporter: Dian Fu
>Assignee: Arvid Heise
>Priority: Major
> Labels: auto-unassigned, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4590=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-07-17T11:43:47.9693015Z [ERROR] Tests run: 12, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 269.399 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> 2020-07-17T11:43:47.9693862Z [ERROR]
> testRecoverCommittedTransaction(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 60.679 s <<< ERROR!
> 2020-07-17T11:43:47.9694737Z org.apache.kafka.common.errors.TimeoutException:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> 2020-07-17T11:43:47.9695376Z Caused by:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] hililiwei commented on pull request #17613: [FLINK-24536][Table SQL/Planner] flink sql support bang equal '!='
hililiwei commented on pull request #17613: URL: https://github.com/apache/flink/pull/17613#issuecomment-969931865 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444298#comment-17444298
]
Yun Gao commented on FLINK-24495:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23586
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24583) ElasticsearchWriterITCase.testWriteOnBulkIntervalFlush timeout on azure
[
https://issues.apache.org/jira/browse/FLINK-24583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444299#comment-17444299
]
Yun Gao commented on FLINK-24583:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=ba53eb01-1462-56a3-8e98-0dd97fbcaab5=2e426bf0-b717-56bb-ab62-d63086457354=12761]
> ElasticsearchWriterITCase.testWriteOnBulkIntervalFlush timeout on azure
> ---
>
> Key: FLINK-24583
> URL: https://issues.apache.org/jira/browse/FLINK-24583
> Project: Flink
> Issue Type: Bug
> Components: Connectors / ElasticSearch
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Assignee: Alexander Preuss
>Priority: Major
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25191=logs=961f8f81-6b52-53df-09f6-7291a2e4af6a=f53023d8-92c3-5d78-ec7e-70c2bf37be20=12452
> {code}
> Oct 18 23:47:27 [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 22.228 s <<< FAILURE! - in
> org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase
> Oct 18 23:47:27 [ERROR] testWriteOnBulkIntervalFlush Time elapsed: 2.032 s
> <<< ERROR!
> Oct 18 23:47:27 java.util.concurrent.TimeoutException: Condition was not met
> in given timeout.
> Oct 18 23:47:27 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:166)
> Oct 18 23:47:27 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:144)
> Oct 18 23:47:27 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:136)
> Oct 18 23:47:27 at
> org.apache.flink.connector.elasticsearch.sink.ElasticsearchWriterITCase.testWriteOnBulkIntervalFlush(ElasticsearchWriterITCase.java:139)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-23944) PulsarSourceITCase.testTaskManagerFailure is instable
[
https://issues.apache.org/jira/browse/FLINK-23944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444296#comment-17444296
]
Yun Gao commented on FLINK-23944:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=24701]
[~syhily] It seems to still happen on the master branch, could you have a
double look~?
> PulsarSourceITCase.testTaskManagerFailure is instable
> -
>
> Key: FLINK-23944
> URL: https://issues.apache.org/jira/browse/FLINK-23944
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Dian Fu
>Assignee: Yufan Sheng
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> [https://dev.azure.com/dianfu/Flink/_build/results?buildId=430=logs=f3dc9b18-b77a-55c1-591e-264c46fe44d1=2d3cd81e-1c37-5c31-0ee4-f5d5cdb9324d]
> It's from my personal azure pipeline, however, I'm pretty sure that I have
> not touched any code related to this.
> {code:java}
> Aug 24 10:44:13 [ERROR] testTaskManagerFailure{TestEnvironment,
> ExternalContext, ClusterControllable}[1] Time elapsed: 258.397 s <<< FAILURE!
> Aug 24 10:44:13 java.lang.AssertionError: Aug 24 10:44:13 Aug 24 10:44:13
> Expected: Records consumed by Flink should be identical to test data and
> preserve the order in split Aug 24 10:44:13 but: Mismatched record at
> position 7: Expected '0W6SzacX7MNL4xLL3BZ8C3ljho4iCydbvxIl' but was
> 'wVi5JaJpNvgkDEOBRC775qHgw0LyRW2HBxwLmfONeEmr' Aug 24 10:44:13 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) Aug 24 10:44:13
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) Aug 24
> 10:44:13 at
> org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase.testTaskManagerFailure(SourceTestSuiteBase.java:271)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-24495) Python installdeps hangs
[
https://issues.apache.org/jira/browse/FLINK-24495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444298#comment-17444298
]
Yun Gao edited comment on FLINK-24495 at 11/16/21, 6:55 AM:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23586]
was (Author: gaoyunhaii):
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23586
> Python installdeps hangs
>
>
> Key: FLINK-24495
> URL: https://issues.apache.org/jira/browse/FLINK-24495
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0
>Reporter: Xintong Song
>Priority: Critical
> Labels: test-stability
> Fix For: 1.15.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=24922=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=23587
> {code}
> Oct 10 02:30:01 py38-cython create: /__w/1/s/flink-python/.tox/py38-cython
> Oct 10 02:30:04 py38-cython installdeps: pytest, apache-beam==2.27.0,
> cython==0.29.16, grpcio>=1.29.0,<2, grpcio-tools>=1.3.5,<=1.14.2,
> apache-flink-libraries
> Oct 10 02:45:22
> ==
> Oct 10 02:45:22 Process produced no output for 900 seconds.
> Oct 10 02:45:22
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-23944) PulsarSourceITCase.testTaskManagerFailure is instable
[
https://issues.apache.org/jira/browse/FLINK-23944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444296#comment-17444296
]
Yun Gao edited comment on FLINK-23944 at 11/16/21, 6:52 AM:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=24701]
Hi [~syhily] It seems to still happen on the master branch, could you have a
double look~?
was (Author: gaoyunhaii):
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26568=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=24701]
[~syhily] It seems to still happen on the master branch, could you have a
double look~?
> PulsarSourceITCase.testTaskManagerFailure is instable
> -
>
> Key: FLINK-23944
> URL: https://issues.apache.org/jira/browse/FLINK-23944
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.14.0
>Reporter: Dian Fu
>Assignee: Yufan Sheng
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.15.0, 1.14.1
>
>
> [https://dev.azure.com/dianfu/Flink/_build/results?buildId=430=logs=f3dc9b18-b77a-55c1-591e-264c46fe44d1=2d3cd81e-1c37-5c31-0ee4-f5d5cdb9324d]
> It's from my personal azure pipeline, however, I'm pretty sure that I have
> not touched any code related to this.
> {code:java}
> Aug 24 10:44:13 [ERROR] testTaskManagerFailure{TestEnvironment,
> ExternalContext, ClusterControllable}[1] Time elapsed: 258.397 s <<< FAILURE!
> Aug 24 10:44:13 java.lang.AssertionError: Aug 24 10:44:13 Aug 24 10:44:13
> Expected: Records consumed by Flink should be identical to test data and
> preserve the order in split Aug 24 10:44:13 but: Mismatched record at
> position 7: Expected '0W6SzacX7MNL4xLL3BZ8C3ljho4iCydbvxIl' but was
> 'wVi5JaJpNvgkDEOBRC775qHgw0LyRW2HBxwLmfONeEmr' Aug 24 10:44:13 at
> org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) Aug 24 10:44:13
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) Aug 24
> 10:44:13 at
> org.apache.flink.connectors.test.common.testsuites.SourceTestSuiteBase.testTaskManagerFailure(SourceTestSuiteBase.java:271)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink-ml] yunfengzhou-hub commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
yunfengzhou-hub commented on a change in pull request #27: URL: https://github.com/apache/flink-ml/pull/27#discussion_r749952767 ## File path: flink-ml-lib/src/main/java/org/apache/flink/ml/param/HasFeaturesCol.java ## @@ -0,0 +1,33 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.param; Review comment: In Alink similar Params like `HasFeatureCol` is in package `org.apache.flink.ml.param.shared.colname`. Shall we also put this and other interfaces into corresponding packages? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24914) Support implicit type conversion in SQL like "select int_field = varchar_field"
[ https://issues.apache.org/jira/browse/FLINK-24914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444294#comment-17444294 ] liwei li commented on FLINK-24914: -- +1 > Support implicit type conversion in SQL like "select int_field = > varchar_field" > --- > > Key: FLINK-24914 > URL: https://issues.apache.org/jira/browse/FLINK-24914 > Project: Flink > Issue Type: Improvement >Reporter: xuyang >Priority: Minor > > Before, "select int_field = varchar_field" will be converted to > "BinaryStringData.equals (int xxx)", so we always get FALSE. > Currently, we temporarily forbid "=" and "<>" between numeric and (var)char > fields in SQL API, just like what have been done in Table API. > In future, we can support the implicit type conversion in Flink SQL and let > this kind of SQL have the correct result. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24916) AvroExternalJarProgramITCase JVM crash on Azure
[
https://issues.apache.org/jira/browse/FLINK-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao updated FLINK-24916:
Labels: test-stability (was: )
> AvroExternalJarProgramITCase JVM crash on Azure
> ---
>
> Key: FLINK-24916
> URL: https://issues.apache.org/jira/browse/FLINK-24916
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.5
>Reporter: Yun Gao
>Priority: Major
> Labels: test-stability
>
> {code:java}
> [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
> [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
> [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
> [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
> [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
> [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
> [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-formats/flink-avro/target &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=1 -XX:+UseG1GC -jar
> /__w/1/s/flink-formats/flink-avro/target/surefire/surefirebooter7435584247231976356.jar
> /__w/1/s/flink-formats/flink-avro/target/surefire
> 2021-11-15T21-36-43_499-jvmRun1 surefire7006265724133532959tmp
> surefire_616019020815389588133tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 127
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26569=logs=3d12d40f-c62d-5ec4-6acc-0efe94cc3e89=5d6e4255-0ea8-5e2a-f52c-c881b7872361=13929]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-20950) SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with "AssertionError"
[
https://issues.apache.org/jira/browse/FLINK-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444292#comment-17444292
]
Yun Gao commented on FLINK-20950:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26569=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=2c7d57b9-7341-5a87-c9af-2cf7cc1a37dc=4549]
[~maguowei] The issue seems to reproduce, could you have another look on it~?
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with
> "AssertionError"
> ---
>
> Key: FLINK-20950
> URL: https://issues.apache.org/jira/browse/FLINK-20950
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.2, 1.13.0, 1.14.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11940=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2]
> {code:java}
> 2021-01-12T16:47:00.7579536Z [ERROR] Failures:
> 2021-01-12T16:47:00.7580061Z [ERROR]
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode:218
> 2021-01-12T16:47:00.7587186Z Expected: iterable over
> ["(895,null,-9223372036854775808)", "(895,null,-9223372036854775808)",
> "(127,null,-9223372036854775808)", "(127,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(161,null,-9223372036854775808)", "(161,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(662,null,-9223372036854775808)", "(662,null,-9223372036854775808)",
> "(822,null,-9223372036854775808)", "(822,null,-9223372036854775808)",
> "(491,null,-9223372036854775808)", "(491,null,-9223372036854775808)",
> "(275,null,-9223372036854775808)", "(275,null,-9223372036854775808)",
> "(122,null,-9223372036854775808)", "(122,null,-9223372036854775808)",
> "(850,null,-9223372036854775808)", "(850,null,-9223372036854775808)",
> "(630,null,-9223372036854775808)", "(630,null,-9223372036854775808)",
> "(682,null,-9223372036854775808)", "(682,null,-9223372036854775808)",
> "(765,null,-9223372036854775808)", "(765,null,-9223372036854775808)",
> "(434,null,-9223372036854775808)", "(434,null,-9223372036854775808)",
> "(970,null,-9223372036854775808)", "(970,null,-9223372036854775808)",
> "(714,null,-9223372036854775808)", "(714,null,-9223372036854775808)",
> "(795,null,-9223372036854775808)", "(795,null,-9223372036854775808)",
> "(288,null,-9223372036854775808)", "(288,null,-9223372036854775808)",
> "(422,null,-9223372036854775808)", "(422,null,-9223372036854775808)"] in any
> order
> 2021-01-12T16:47:00.7591663Z but: Not matched: "end of input"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24911) Enable line numbers in SQL Client
[
https://issues.apache.org/jira/browse/FLINK-24911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-24911:
Component/s: Table SQL / Client
> Enable line numbers in SQL Client
> -
>
> Key: FLINK-24911
> URL: https://issues.apache.org/jira/browse/FLINK-24911
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Client
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> Should be enabled/disabled via property
> {{sql-client.prompt.show-line-numbers}}
> Also add widget to make it possible to toggle with a key-stroke
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Reopened] (FLINK-20950) SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with "AssertionError"
[
https://issues.apache.org/jira/browse/FLINK-20950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yun Gao reopened FLINK-20950:
-
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode test failed with
> "AssertionError"
> ---
>
> Key: FLINK-20950
> URL: https://issues.apache.org/jira/browse/FLINK-20950
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.12.2, 1.13.0, 1.14.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: auto-deprioritized-critical, auto-unassigned,
> test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11940=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2]
> {code:java}
> 2021-01-12T16:47:00.7579536Z [ERROR] Failures:
> 2021-01-12T16:47:00.7580061Z [ERROR]
> SinkITCase.writerAndGlobalCommitterExecuteInStreamingMode:218
> 2021-01-12T16:47:00.7587186Z Expected: iterable over
> ["(895,null,-9223372036854775808)", "(895,null,-9223372036854775808)",
> "(127,null,-9223372036854775808)", "(127,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(161,null,-9223372036854775808)", "(161,null,-9223372036854775808)",
> "(148,null,-9223372036854775808)", "(148,null,-9223372036854775808)",
> "(662,null,-9223372036854775808)", "(662,null,-9223372036854775808)",
> "(822,null,-9223372036854775808)", "(822,null,-9223372036854775808)",
> "(491,null,-9223372036854775808)", "(491,null,-9223372036854775808)",
> "(275,null,-9223372036854775808)", "(275,null,-9223372036854775808)",
> "(122,null,-9223372036854775808)", "(122,null,-9223372036854775808)",
> "(850,null,-9223372036854775808)", "(850,null,-9223372036854775808)",
> "(630,null,-9223372036854775808)", "(630,null,-9223372036854775808)",
> "(682,null,-9223372036854775808)", "(682,null,-9223372036854775808)",
> "(765,null,-9223372036854775808)", "(765,null,-9223372036854775808)",
> "(434,null,-9223372036854775808)", "(434,null,-9223372036854775808)",
> "(970,null,-9223372036854775808)", "(970,null,-9223372036854775808)",
> "(714,null,-9223372036854775808)", "(714,null,-9223372036854775808)",
> "(795,null,-9223372036854775808)", "(795,null,-9223372036854775808)",
> "(288,null,-9223372036854775808)", "(288,null,-9223372036854775808)",
> "(422,null,-9223372036854775808)", "(422,null,-9223372036854775808)"] in any
> order
> 2021-01-12T16:47:00.7591663Z but: Not matched: "end of input"
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24910) Propagate the Calcite parser config to SQL Client
[ https://issues.apache.org/jira/browse/FLINK-24910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin updated FLINK-24910: Component/s: Table SQL / Client > Propagate the Calcite parser config to SQL Client > - > > Key: FLINK-24910 > URL: https://issues.apache.org/jira/browse/FLINK-24910 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Client >Reporter: Sergey Nuyanzin >Priority: Major > > It's required to get Dialect specific info like keywords, sql quote identifier -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24909) SQL syntax highlighting in SQL Client
[
https://issues.apache.org/jira/browse/FLINK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-24909:
Component/s: Table SQL / Client
> SQL syntax highlighting in SQL Client
> -
>
> Key: FLINK-24909
> URL: https://issues.apache.org/jira/browse/FLINK-24909
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Client
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> What to hightlight: keywords, quoted strings, sql identifier quoted string,
> line comments, block comments, hints.
> Property {{sql-client.color-schema}} to set current highlighting schema
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-24916) AvroExternalJarProgramITCase JVM crash on Azure
Yun Gao created FLINK-24916:
---
Summary: AvroExternalJarProgramITCase JVM crash on Azure
Key: FLINK-24916
URL: https://issues.apache.org/jira/browse/FLINK-24916
Project: Flink
Issue Type: Bug
Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.12.5
Reporter: Yun Gao
{code:java}
[ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
[ERROR] at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
[ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
[ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
[ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
[ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[ERROR] Caused by:
org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /__w/1/s/flink-formats/flink-avro/target &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
-Dmvn.forkNumber=1 -XX:+UseG1GC -jar
/__w/1/s/flink-formats/flink-avro/target/surefire/surefirebooter7435584247231976356.jar
/__w/1/s/flink-formats/flink-avro/target/surefire
2021-11-15T21-36-43_499-jvmRun1 surefire7006265724133532959tmp
surefire_616019020815389588133tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 127
{code}
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26569=logs=3d12d40f-c62d-5ec4-6acc-0efe94cc3e89=5d6e4255-0ea8-5e2a-f52c-c881b7872361=13929]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-24916) AvroExternalJarProgramITCase JVM crash on Azure
[
https://issues.apache.org/jira/browse/FLINK-24916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444291#comment-17444291
]
Yun Gao commented on FLINK-24916:
-
The maven log does not show additional exceptions.
> AvroExternalJarProgramITCase JVM crash on Azure
> ---
>
> Key: FLINK-24916
> URL: https://issues.apache.org/jira/browse/FLINK-24916
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.5
>Reporter: Yun Gao
>Priority: Major
>
> {code:java}
> [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
> [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
> [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
> [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
> [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
> [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
> [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-formats/flink-avro/target &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=1 -XX:+UseG1GC -jar
> /__w/1/s/flink-formats/flink-avro/target/surefire/surefirebooter7435584247231976356.jar
> /__w/1/s/flink-formats/flink-avro/target/surefire
> 2021-11-15T21-36-43_499-jvmRun1 surefire7006265724133532959tmp
> surefire_616019020815389588133tmp
> [ERROR] Error occurred in starting fork, check output in log
> [ERROR] Process Exit Code: 127
> {code}
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26569=logs=3d12d40f-c62d-5ec4-6acc-0efe94cc3e89=5d6e4255-0ea8-5e2a-f52c-c881b7872361=13929]
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24908) Improve SQL Error description for SQL Client
[
https://issues.apache.org/jira/browse/FLINK-24908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin updated FLINK-24908:
Component/s: Table SQL / Client
> Improve SQL Error description for SQL Client
>
>
> Key: FLINK-24908
> URL: https://issues.apache.org/jira/browse/FLINK-24908
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Client
>Reporter: Sergey Nuyanzin
>Priority: Major
>
> For instance for misprint like
> {code:sql}
> ELECT 1;
> {code}
> The error looks like
> {noformat}
> ELECT 1;
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.calcite.runtime.CalciteException: Non-query expression encountered
> in illegal context
> {noformat}
> At the same time ParseException wrapped by CalciteException contains position
> of a problem word
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-24893) SQL Client prompt customisation
[ https://issues.apache.org/jira/browse/FLINK-24893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin updated FLINK-24893: Component/s: Table SQL / Client > SQL Client prompt customisation > --- > > Key: FLINK-24893 > URL: https://issues.apache.org/jira/browse/FLINK-24893 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Client >Reporter: Sergey Nuyanzin >Priority: Major > > Also the details are here > https://cwiki.apache.org/confluence/display/FLINK/FLIP-189%3A+SQL+Client+Usability+Improvements#FLIP189:SQLClientUsabilityImprovements-Supportedpromptoptions(bothforleftandrightprompts) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17754: [FLINK-24861][connector][jdbc] Fix false cache lookup for empty data
flinkbot edited a comment on pull request #17754: URL: https://github.com/apache/flink/pull/17754#issuecomment-965189525 ## CI report: * 6a2564f49bc2fb8d3f82190184bea34fad523e2b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26535) * b3703f43ebbe0c33d7ca3a607b7cd6fb49169e5b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26584) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17749: [FLINK-24758][Connectors / FileSystem] filesystem sink: add partitiontime-extractor.formatter-pattern to allow user to speify DateTi
flinkbot edited a comment on pull request #17749: URL: https://github.com/apache/flink/pull/17749#issuecomment-965029957 ## CI report: * 7274c72e7c23ac670a39bf6bac0eacbe881db110 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26557) * c47537f747d197cd3b6ed565ba479c11f7175a40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26583) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17749: [FLINK-24758][Connectors / FileSystem] filesystem sink: add partitiontime-extractor.formatter-pattern to allow user to speify DateTi
flinkbot edited a comment on pull request #17749: URL: https://github.com/apache/flink/pull/17749#issuecomment-965029957 ## CI report: * 7274c72e7c23ac670a39bf6bac0eacbe881db110 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26557) * c47537f747d197cd3b6ed565ba479c11f7175a40 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17754: [FLINK-24861][connector][jdbc] Fix false cache lookup for empty data
flinkbot edited a comment on pull request #17754: URL: https://github.com/apache/flink/pull/17754#issuecomment-965189525 ## CI report: * 6a2564f49bc2fb8d3f82190184bea34fad523e2b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26535) * b3703f43ebbe0c33d7ca3a607b7cd6fb49169e5b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaurav726 commented on a change in pull request #17754: [FLINK-24861][connector][jdbc] Fix false cache lookup for empty data
gaurav726 commented on a change in pull request #17754:
URL: https://github.com/apache/flink/pull/17754#discussion_r749949235
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcRowDataLookupFunction.java
##
@@ -146,10 +148,12 @@ public void eval(Object... keys) {
if (cache != null) {
List cachedRows = cache.getIfPresent(keyRow);
if (cachedRows != null) {
-for (RowData cachedRow : cachedRows) {
-collect(cachedRow);
+if (!cachedRows.isEmpty() || !excludeEmptyQueryResult) {
+for (RowData cachedRow : cachedRows) {
Review comment:
@wenlong88 added test cases, please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] gaoyunhaii commented on a change in pull request #27: [FLINK-24810] Add Estimator and Model for the k-means clustering algorithm
gaoyunhaii commented on a change in pull request #27:
URL: https://github.com/apache/flink-ml/pull/27#discussion_r749940719
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/clustering/kmeans/KMeansModel.java
##
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.clustering.kmeans;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.common.state.ListState;
+import org.apache.flink.api.common.state.ListStateDescriptor;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.connector.source.Source;
+import org.apache.flink.api.java.typeutils.ObjectArrayTypeInfo;
+import org.apache.flink.api.java.typeutils.RowTypeInfo;
+import org.apache.flink.connector.file.sink.FileSink;
+import org.apache.flink.connector.file.src.FileSource;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.ml.api.core.Model;
+import org.apache.flink.ml.common.datastream.TableUtils;
+import org.apache.flink.ml.distance.DistanceMeasure;
+import org.apache.flink.ml.linalg.DenseVector;
+import org.apache.flink.ml.linalg.typeinfo.DenseVectorTypeInfo;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.runtime.state.StateInitializationContext;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.bucketassigners.BasePathBucketAssigner;
+import
org.apache.flink.streaming.api.functions.sink.filesystem.rollingpolicies.OnCheckpointRollingPolicy;
+import org.apache.flink.streaming.api.operators.AbstractStreamOperator;
+import org.apache.flink.streaming.api.operators.TwoInputStreamOperator;
+import org.apache.flink.streaming.runtime.streamrecord.StreamRecord;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+
+import org.apache.commons.collections.IteratorUtils;
+import org.apache.commons.lang3.ArrayUtils;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/** A Model which clusters data into k clusters using the model data computed
by {@link KMeans}. */
+public class KMeansModel implements Model,
KMeansModelParams {
+private final Map, Object> paramMap = new HashMap<>();
+private Table centroidsTable;
+
+public KMeansModel() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public KMeansModel setModelData(Table... inputs) {
+centroidsTable = inputs[0];
+return this;
+}
+
+@Override
+public Table[] getModelData() {
+return new Table[] {centroidsTable};
+}
+
+@Override
+public Table[] transform(Table... inputs) {
+StreamTableEnvironment tEnv =
+(StreamTableEnvironment) ((TableImpl)
inputs[0]).getTableEnvironment();
+DataStream centroids =
+tEnv.toDataStream(centroidsTable).map(row -> (DenseVector[])
row.getField("f0"));
+
+String featureCol = getFeaturesCol();
+String predictionCol = getPredictionCol();
+DistanceMeasure distanceMeasure =
DistanceMeasure.getInstance(getDistanceMeasure());
+
+RowTypeInfo inputTypeInfo =
TableUtils.getRowTypeInfo(inputs[0].getResolvedSchema());
+RowTypeInfo outputTypeInfo =
+new RowTypeInfo(
+ArrayUtils.addAll(inputTypeInfo.getFieldTypes(),
Types.INT),
+ArrayUtils.addAll(inputTypeInfo.getFieldNames(),
predictionCol));
+
+DataStream input = tEnv.toDataStream(inputs[0]);
+DataStream output =
+input.connect(centroids.broadcast())
+.transform(
+"SelectNearestCentroid",
+outputTypeInfo,
+
[GitHub] [flink] RocMarshal commented on a change in pull request #17802: [FLINK-24635][examples] Clean up flink-examples
RocMarshal commented on a change in pull request #17802: URL: https://github.com/apache/flink/pull/17802#discussion_r749942475 ## File path: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/async/util/SimpleSource.java ## @@ -0,0 +1,59 @@ +package org.apache.flink.streaming.examples.async.util; Review comment: missing of licenses ## File path: flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/windowing/util/CarSource.java ## @@ -0,0 +1,59 @@ +package org.apache.flink.streaming.examples.windowing.util; Review comment: lack of licenses -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
RocMarshal edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-966782262 @twalthr @sjwiesman @Airblader @AHeise @tisonkun @fsk119 Excuse me, Could you help me to check it if you have free time ? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17387: [FLINK-24864][metrics] Release TaskManagerJobMetricGroup with the last slot rather than task
flinkbot edited a comment on pull request #17387: URL: https://github.com/apache/flink/pull/17387#issuecomment-930245327 ## CI report: * 1a458e189a85566571b071fd0bc54df9317d2db3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=26575) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal edited a comment on pull request #16962: [FLINK-15352][connector-jdbc] Develop MySQLCatalog to connect Flink with MySQL tables and ecosystem.
RocMarshal edited a comment on pull request #16962: URL: https://github.com/apache/flink/pull/16962#issuecomment-966782262 @twalthr @sjwiesman @Airblader @AHeise @tisonkun @gaoyunhaii Excuse me, Could you help me to check it if you have free time ? Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] lindong28 commented on a change in pull request #32: [FLINK-24817] Add Naive Bayes implementation
lindong28 commented on a change in pull request #32:
URL: https://github.com/apache/flink-ml/pull/32#discussion_r749901463
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams extends
Review comment:
Could we add a parameter like modelType here? I suppose we will
eventually expand NaiveBayes to support all modelTypes currently supported by
Spark.
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training process.
+ */
+public interface NaiveBayesParams extends
+HasFeatureCols,
+HasLabelCol,
+HasPredictionCol {
+Param SMOOTHING =
+new DoubleParam(
+"smoothing",
+"the smoothing factor",
+0.0,
Review comment:
The default value in Spark is 1.0. Do you think 1.0 would be a better
default value here?
##
File path:
flink-ml-lib/src/main/java/org/apache/flink/ml/classification/naivebayes/NaiveBayesParams.java
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.classification.naivebayes;
+
+import org.apache.flink.ml.param.DoubleParam;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.param.ParamValidators;
+import org.apache.flink.ml.param.shared.colname.HasFeatureCols;
+import org.apache.flink.ml.param.shared.colname.HasLabelCol;
+import org.apache.flink.ml.param.shared.colname.HasPredictionCol;
+
+/**
+ * Parameters of naive bayes training
[jira] [Commented] (FLINK-24914) Support implicit type conversion in SQL like "select int_field = varchar_field"
[ https://issues.apache.org/jira/browse/FLINK-24914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444282#comment-17444282 ] xuyang commented on FLINK-24914: Hi, [~qingru zhang] . In a sense, it is, but FLIP-154 have not been discussed enough I think. > Support implicit type conversion in SQL like "select int_field = > varchar_field" > --- > > Key: FLINK-24914 > URL: https://issues.apache.org/jira/browse/FLINK-24914 > Project: Flink > Issue Type: Improvement >Reporter: xuyang >Priority: Minor > > Before, "select int_field = varchar_field" will be converted to > "BinaryStringData.equals (int xxx)", so we always get FALSE. > Currently, we temporarily forbid "=" and "<>" between numeric and (var)char > fields in SQL API, just like what have been done in Table API. > In future, we can support the implicit type conversion in Flink SQL and let > this kind of SQL have the correct result. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] JingsongLi commented on a change in pull request #17655: [FLINK-24728][tests] Add tests to ensure SQL file sink closes all created files
JingsongLi commented on a change in pull request #17655:
URL: https://github.com/apache/flink/pull/17655#discussion_r749933893
##
File path:
flink-core/src/test/java/org/apache/flink/testutils/TestFileSystem.java
##
@@ -37,28 +43,45 @@
public static final String SCHEME = "test";
-private static int streamOpenCounter;
+// number of (input) stream opened
+private static final AtomicInteger streamOpenCounter = new
AtomicInteger(0);
+
+// current number of created, unclosed (output) stream
+private static final Map currentUnclosedOutputStream = new
ConcurrentHashMap<>();
public static int getNumtimeStreamOpened() {
-return streamOpenCounter;
+return streamOpenCounter.get();
+}
+
+public static Map getCurrentUnclosedOutputStream() {
+return currentUnclosedOutputStream;
}
-public static void resetStreamOpenCounter() {
-streamOpenCounter = 0;
+public static void resetStreamCounter() {
Review comment:
Can we just `reset(Path)`? I think maybe there are some unstable cases
if just clear all counters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
JingsongLi commented on pull request #17520: URL: https://github.com/apache/flink/pull/17520#issuecomment-969890963 @slinkydeveloper @JingGe @fapaul Do you have other comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24767) A keyBy following countWindow does not preserve order within the same partition
[
https://issues.apache.org/jira/browse/FLINK-24767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444280#comment-17444280
]
Guowei Ma commented on FLINK-24767:
---
I think it might be that in the Batch execution mode FLINK will "sort" the Key
and this might be an unstable sort.
> A keyBy following countWindow does not preserve order within the same
> partition
> ---
>
> Key: FLINK-24767
> URL: https://issues.apache.org/jira/browse/FLINK-24767
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.13.3
>Reporter: Lee Y S
>Priority: Major
>
> I wrote a simple test of the countWindow method (in Kotlin) as below
> {code:java}
> import org.apache.flink.api.common.RuntimeExecutionMode
> import org.apache.flink.api.common.eventtime.WatermarkStrategy
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment
> import kotlin.random.Randomobject
> CountWindowTest {
> @JvmStatic
> fun main(args: Array) {
> val env = StreamExecutionEnvironment.getExecutionEnvironment()
> env.setRuntimeMode(RuntimeExecutionMode.BATCH)
> val rand = Random(0)
> val data = (0..1000).map { Pair(rand.nextInt(10), it) }
> env.fromCollection(data).assignTimestampsAndWatermarks(
> WatermarkStrategy.forMonotonousTimestamps>()
> .withTimestampAssigner { e, _ -> e.second.toLong() })
> .keyBy { it.first }
> .countWindow(3L, 1)
> .reduce { a, b -> b }
> .keyBy { it.first }
> .filter { it.first == 5 }
> .print()
> env.execute()
> }
> }
> {code}
> The beginning of the output is as below
> 12> (5, 184)
> 12> (5, 18)
> 12> (5, 29)
> 12> (5, 37)
> 12> (5, 38)
> 12> (5, 112)
> 12> (5, 131)
> The first line (5, 184) is not in order from the rest.
> The problem disappears if I remove the keyBy after the reduce or use stream
> mode instead of batch mode.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Comment Edited] (FLINK-23466) UnalignedCheckpointITCase hangs on Azure
[
https://issues.apache.org/jira/browse/FLINK-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444279#comment-17444279
]
Yingjie Cao edited comment on FLINK-23466 at 11/16/21, 5:59 AM:
Nov 10 16:13:03 Starting
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase#execute[pipeline
with mixed channels, p = 20, timeout = 0, buffersPerChannel = 1].
>From the log, we can see this case hangs. I guess this seems a new issue which
>is different from the one reported in this ticket. From the stack, it seems
>there is something wrong with the checkpoint coordinator, the following thread
>locked 0x87db4fb8:
{code:java}
2021-11-10T17:14:21.0899474Z Nov 10 17:14:21 "jobmanager-io-thread-2" #12984
daemon prio=5 os_prio=0 tid=0x7f12e000b800 nid=0x3fb6 runnable
[0x7f0fcd6d4000]
2021-11-10T17:14:21.0899924Z Nov 10 17:14:21java.lang.Thread.State: RUNNABLE
2021-11-10T17:14:21.0900300Z Nov 10 17:14:21at
java.util.HashMap$TreeNode.balanceDeletion(HashMap.java:2338)
2021-11-10T17:14:21.0900745Z Nov 10 17:14:21at
java.util.HashMap$TreeNode.removeTreeNode(HashMap.java:2112)
2021-11-10T17:14:21.0901146Z Nov 10 17:14:21at
java.util.HashMap.removeNode(HashMap.java:840)
2021-11-10T17:14:21.0901577Z Nov 10 17:14:21at
java.util.LinkedHashMap.afterNodeInsertion(LinkedHashMap.java:301)
2021-11-10T17:14:21.0902002Z Nov 10 17:14:21at
java.util.HashMap.putVal(HashMap.java:664)
2021-11-10T17:14:21.0902531Z Nov 10 17:14:21at
java.util.HashMap.putMapEntries(HashMap.java:515)
2021-11-10T17:14:21.0902931Z Nov 10 17:14:21at
java.util.HashMap.putAll(HashMap.java:785)
2021-11-10T17:14:21.0903429Z Nov 10 17:14:21at
org.apache.flink.runtime.checkpoint.ExecutionAttemptMappingProvider.getVertex(ExecutionAttemptMappingProvider.java:60)
2021-11-10T17:14:21.0904060Z Nov 10 17:14:21at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.reportStats(CheckpointCoordinator.java:1867)
2021-11-10T17:14:21.0904686Z Nov 10 17:14:21at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1152)
2021-11-10T17:14:21.0905372Z Nov 10 17:14:21- locked <0x87db4fb8>
(a java.lang.Object)
2021-11-10T17:14:21.0905895Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
2021-11-10T17:14:21.0906493Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler$$Lambda$1368/705813936.accept(Unknown
Source)
2021-11-10T17:14:21.0907086Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
2021-11-10T17:14:21.0907698Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler$$Lambda$1369/1447418658.run(Unknown
Source)
2021-11-10T17:14:21.0908210Z Nov 10 17:14:21at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-11-10T17:14:21.0908735Z Nov 10 17:14:21at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-11-10T17:14:21.0909333Z Nov 10 17:14:21at
java.lang.Thread.run(Thread.java:748) {code}
But other thread is waiting for the lock. I am not familiar with these logics
and not sure if this is in the right state. Could anyone who is familiar with
these logics take a look?
BTW, concurrent access of HashMap may cause infinite loop，I see in the stack
that there are multiple threads are accessing HashMap, though I am not sure if
they are the same instance.
was (Author: kevin.cyj):
Nov 10 16:13:03 Starting
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase#execute[pipeline
with mixed channels, p = 20, timeout = 0, buffersPerChannel = 1].
>From the log, we can see this case hangs. I guess this seems a new issue. From
>the stack, it seems there is something wrong with the checkpoint coordinator,
>the following thread locked 0x87db4fb8:
{code:java}
2021-11-10T17:14:21.0899474Z Nov 10 17:14:21 "jobmanager-io-thread-2" #12984
daemon prio=5 os_prio=0 tid=0x7f12e000b800 nid=0x3fb6 runnable
[0x7f0fcd6d4000]
2021-11-10T17:14:21.0899924Z Nov 10 17:14:21java.lang.Thread.State: RUNNABLE
2021-11-10T17:14:21.0900300Z Nov 10 17:14:21at
java.util.HashMap$TreeNode.balanceDeletion(HashMap.java:2338)
2021-11-10T17:14:21.0900745Z Nov 10 17:14:21at
java.util.HashMap$TreeNode.removeTreeNode(HashMap.java:2112)
2021-11-10T17:14:21.0901146Z Nov 10 17:14:21at
java.util.HashMap.removeNode(HashMap.java:840)
2021-11-10T17:14:21.0901577Z Nov 10 17:14:21at
java.util.LinkedHashMap.afterNodeInsertion(LinkedHashMap.java:301)
2021-11-10T17:14:21.0902002Z Nov 10 17:14:21at
java.util.HashMap.putVal(HashMap.java:664)
2021-11-10T17:14:21.0902531Z Nov 10 17:14:21at
[jira] [Commented] (FLINK-23466) UnalignedCheckpointITCase hangs on Azure
[
https://issues.apache.org/jira/browse/FLINK-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444279#comment-17444279
]
Yingjie Cao commented on FLINK-23466:
-
Nov 10 16:13:03 Starting
org.apache.flink.test.checkpointing.UnalignedCheckpointITCase#execute[pipeline
with mixed channels, p = 20, timeout = 0, buffersPerChannel = 1].
>From the log, we can see this case hangs. I guess this seems a new issue. From
>the stack, it seems there is something wrong with the checkpoint coordinator,
>the following thread locked 0x87db4fb8:
{code:java}
2021-11-10T17:14:21.0899474Z Nov 10 17:14:21 "jobmanager-io-thread-2" #12984
daemon prio=5 os_prio=0 tid=0x7f12e000b800 nid=0x3fb6 runnable
[0x7f0fcd6d4000]
2021-11-10T17:14:21.0899924Z Nov 10 17:14:21java.lang.Thread.State: RUNNABLE
2021-11-10T17:14:21.0900300Z Nov 10 17:14:21at
java.util.HashMap$TreeNode.balanceDeletion(HashMap.java:2338)
2021-11-10T17:14:21.0900745Z Nov 10 17:14:21at
java.util.HashMap$TreeNode.removeTreeNode(HashMap.java:2112)
2021-11-10T17:14:21.0901146Z Nov 10 17:14:21at
java.util.HashMap.removeNode(HashMap.java:840)
2021-11-10T17:14:21.0901577Z Nov 10 17:14:21at
java.util.LinkedHashMap.afterNodeInsertion(LinkedHashMap.java:301)
2021-11-10T17:14:21.0902002Z Nov 10 17:14:21at
java.util.HashMap.putVal(HashMap.java:664)
2021-11-10T17:14:21.0902531Z Nov 10 17:14:21at
java.util.HashMap.putMapEntries(HashMap.java:515)
2021-11-10T17:14:21.0902931Z Nov 10 17:14:21at
java.util.HashMap.putAll(HashMap.java:785)
2021-11-10T17:14:21.0903429Z Nov 10 17:14:21at
org.apache.flink.runtime.checkpoint.ExecutionAttemptMappingProvider.getVertex(ExecutionAttemptMappingProvider.java:60)
2021-11-10T17:14:21.0904060Z Nov 10 17:14:21at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.reportStats(CheckpointCoordinator.java:1867)
2021-11-10T17:14:21.0904686Z Nov 10 17:14:21at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1152)
2021-11-10T17:14:21.0905372Z Nov 10 17:14:21- locked <0x87db4fb8>
(a java.lang.Object)
2021-11-10T17:14:21.0905895Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89)
2021-11-10T17:14:21.0906493Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler$$Lambda$1368/705813936.accept(Unknown
Source)
2021-11-10T17:14:21.0907086Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119)
2021-11-10T17:14:21.0907698Z Nov 10 17:14:21at
org.apache.flink.runtime.scheduler.ExecutionGraphHandler$$Lambda$1369/1447418658.run(Unknown
Source)
2021-11-10T17:14:21.0908210Z Nov 10 17:14:21at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2021-11-10T17:14:21.0908735Z Nov 10 17:14:21at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2021-11-10T17:14:21.0909333Z Nov 10 17:14:21at
java.lang.Thread.run(Thread.java:748) {code}
But other thread is waiting for the lock. I am not familiar with these logics
and not sure if this is in the right state. Could anyone who is familiar with
these logics take a look?
BTW, concurrent access of HashMap may cause infinite loop，I see in the stack
that there are multiple threads are accessing HashMap, though I am not sure if
they are the same instance.
> UnalignedCheckpointITCase hangs on Azure
>
>
> Key: FLINK-23466
> URL: https://issues.apache.org/jira/browse/FLINK-23466
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.14.0
>Reporter: Dawid Wysakowicz
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.14.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=20813=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=2ef0effc-1da1-50e5-c2bd-aab434b1c5b7=16016
> The problem is the buffer listener will be removed from the listener queue
> when notified and then it will be added to the listener queue again if it
> needs more buffers. However, if some buffers are recycled meanwhile, the
> buffer listener will not be notified of the available buffers. For example:
> 1. Thread 1 calls LocalBufferPool#recycle().
> 2. Thread 1 reaches LocalBufferPool#fireBufferAvailableNotification() and
> listener.notifyBufferAvailable() is invoked, but Thread 1 sleeps before
> acquiring the lock to registeredListeners.add(listener).
> 3. Thread 2 is being woken up as a result of notifyBufferAvailable()
> call. It takes the buffer, but it needs more buffers.
> 4. Other
