[jira] [Commented] (FLINK-33220) PyFlink support for Datagen connector
[ https://issues.apache.org/jira/browse/FLINK-33220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773534#comment-17773534 ] Liu Chong commented on FLINK-33220: --- we've got the code ready for adding datagen to pyflink, would you like to give some comments if we should move forward and submit the PR? [~dianfu] > PyFlink support for Datagen connector > - > > Key: FLINK-33220 > URL: https://issues.apache.org/jira/browse/FLINK-33220 > Project: Flink > Issue Type: New Feature > Components: API / Python >Reporter: Liu Chong >Priority: Minor > > This is a simple Jira to propose the support of Datagen in PyFlink datastream > API as a built-in source connector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33197) PyFlink support for ByteArraySchema
[ https://issues.apache.org/jira/browse/FLINK-33197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773536#comment-17773536 ] Liu Chong commented on FLINK-33197: --- we've got the code ready for adding ByteArraySchema to pyflink, would you like to give some comments if we should move forward and submit the PR? [~dianfu] > PyFlink support for ByteArraySchema > --- > > Key: FLINK-33197 > URL: https://issues.apache.org/jira/browse/FLINK-33197 > Project: Flink > Issue Type: New Feature > Components: API / Python >Affects Versions: 1.17.0 >Reporter: Liu Chong >Priority: Minor > > Currently in Python Flink API, when reading messages from a Kafka source, > only SimpleStringSchema is available. > If the data is in arbitary binary format(e.g. marshalled Protocol Buffer msg) > it may not be decodable with the default 'utf-8' encoding. > There's currently a workaround which is to manually set the encoding to > 'ISO-8859-1' which supports all possible byte combinations. > However this is not an elegant solution. > We should support ByteArraySchema which outputs a raw byte array for > subsequent unmarshalling. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33227) Flink sql cep support 'notFollowedBy' semantics
[ https://issues.apache.org/jira/browse/FLINK-33227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaoran updated FLINK-33227: Description: Currently, the cep mode of the Flink API can support next, notNext, followedBy, followedByAny, and notFollowedBy semantics, but Flink SQL only supports next semantics. The remaining notNext and followedBy semantics are implemented by other alternatives, while the notFollowedBy semantics are not currently implemented. At present, this semantics is generally implemented in business scenarios, such as judging that a user has placed an order within 15 minutes without paying. Therefore, I suggest to provide new functionality to support notFollowedBy in sql mode, along with the other three semantics (was: Currently, the cep mode of the Flink API can support next, notNext, followedBy, followedByAny, and notFollowedBy semantics, but Flink SQL only supports next semantics. The remaining notNext and followedBy semantics are implemented by other alternatives, while the notFollowedBy semantics are not currently implemented. At present, this semantics is generally implemented in business scenarios, such as judging that a user has placed an order within 15 minutes without paying. Therefore, I want to provide new functionality to support notFollowedBy in sql mode, along with the other three semantics) > Flink sql cep support 'notFollowedBy' semantics > --- > > Key: FLINK-33227 > URL: https://issues.apache.org/jira/browse/FLINK-33227 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.16.0 >Reporter: xiaoran >Priority: Major > > Currently, the cep mode of the Flink API can support next, notNext, > followedBy, followedByAny, and notFollowedBy semantics, but Flink SQL only > supports next semantics. The remaining notNext and followedBy semantics are > implemented by other alternatives, while the notFollowedBy semantics are not > currently implemented. At present, this semantics is generally implemented in > business scenarios, such as judging that a user has placed an order within 15 > minutes without paying. Therefore, I suggest to provide new functionality to > support notFollowedBy in sql mode, along with the other three semantics -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33227) Flink sql cep support 'notFollowedBy' semantics
xiaoran created FLINK-33227: --- Summary: Flink sql cep support 'notFollowedBy' semantics Key: FLINK-33227 URL: https://issues.apache.org/jira/browse/FLINK-33227 Project: Flink Issue Type: New Feature Components: Table SQL / API Affects Versions: 1.16.0 Reporter: xiaoran Currently, the cep mode of the Flink API can support next, notNext, followedBy, followedByAny, and notFollowedBy semantics, but Flink SQL only supports next semantics. The remaining notNext and followedBy semantics are implemented by other alternatives, while the notFollowedBy semantics are not currently implemented. At present, this semantics is generally implemented in business scenarios, such as judging that a user has placed an order within 15 minutes without paying. Therefore, I want to provide new functionality to support notFollowedBy in sql mode, along with the other three semantics -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-24024) Support session Window TVF
[ https://issues.apache.org/jira/browse/FLINK-24024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773517#comment-17773517 ] xuyang commented on FLINK-24024: Hi, [~Sergey Nuyanzin] [~twalthr] ，I did a basic poc on my branch here '[https://github.com/xuyangzhong/flink/tree/FLINK-24024'|https://github.com/xuyangzhong/flink/tree/FLINK-24024'.] . In this branch, at the grammar part and planner part the session window tvf will be supported. So I think the current calcite version 1.32.0 has met the requirements to support session window tvf. WDYT? > Support session Window TVF > --- > > Key: FLINK-24024 > URL: https://issues.apache.org/jira/browse/FLINK-24024 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Jing Zhang >Assignee: Sergey Nuyanzin >Priority: Major > Labels: pull-request-available, stale-assigned > > > # Fix calcite syntax CALCITE-4337 > # Introduce session window TVF in Flink > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-25593][table-planner] Skip redundant scan while partition table push down contains none-existent partition [flink]
swuferhong commented on code in PR #23423:
URL: https://github.com/apache/flink/pull/23423#discussion_r1351218921
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushPartitionIntoTableSourceScanRule.java:
##
@@ -176,6 +177,14 @@ public void onMatch(RelOptRuleCall call) {
defaultPruner,
allPredicates._1(),
inputFieldNames);
+
+// If remaining partitions are empty, it means that there are no
partitions are selected
+// after partition prune. We can directly optimize the RelNode to
Empty FlinkLogicalValues.
+if (remainingPartitions.isEmpty()) {
+RelNode emptyValue = relBuilder.push(filter).empty().build();
Review Comment:
> Can it be `relBuilder.empty().build()`?
It must need a leaf node, otherwise the `NoSuchElementException` will be
thrown.
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushPartitionIntoTableSourceScanRule.java:
##
@@ -176,6 +177,14 @@ public void onMatch(RelOptRuleCall call) {
defaultPruner,
allPredicates._1(),
inputFieldNames);
+
+// If remaining partitions are empty, it means that there are no
partitions are selected
+// after partition prune. We can directly optimize the RelNode to
Empty FlinkLogicalValues.
+if (remainingPartitions.isEmpty()) {
+RelNode emptyValue = relBuilder.push(filter).empty().build();
+call.transformTo(emptyValue);
Review Comment:
> return after this line?
Done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-25593][table-planner] Skip redundant scan while partition table push down contains none-existent partition [flink]
luoyuxia commented on code in PR #23423:
URL: https://github.com/apache/flink/pull/23423#discussion_r1351171194
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushPartitionIntoTableSourceScanRule.java:
##
@@ -176,6 +177,14 @@ public void onMatch(RelOptRuleCall call) {
defaultPruner,
allPredicates._1(),
inputFieldNames);
+
+// If remaining partitions are empty, it means that there are no
partitions are selected
+// after partition prune. We can directly optimize the RelNode to
Empty FlinkLogicalValues.
+if (remainingPartitions.isEmpty()) {
+RelNode emptyValue = relBuilder.push(filter).empty().build();
Review Comment:
Can it be `relBuilder.empty().build()`?
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/PushPartitionIntoTableSourceScanRule.java:
##
@@ -176,6 +177,14 @@ public void onMatch(RelOptRuleCall call) {
defaultPruner,
allPredicates._1(),
inputFieldNames);
+
+// If remaining partitions are empty, it means that there are no
partitions are selected
+// after partition prune. We can directly optimize the RelNode to
Empty FlinkLogicalValues.
+if (remainingPartitions.isEmpty()) {
+RelNode emptyValue = relBuilder.push(filter).empty().build();
+call.transformTo(emptyValue);
Review Comment:
return after this line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [docs] Update Vendor Solutions [flink]
JingGe merged PR #23482: URL: https://github.com/apache/flink/pull/23482 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33226) Forbid to drop current database
Shengkai Fang created FLINK-33226:
-
Summary: Forbid to drop current database
Key: FLINK-33226
URL: https://issues.apache.org/jira/browse/FLINK-33226
Project: Flink
Issue Type: Improvement
Reporter: Shengkai Fang
PG or MySql both doesn't support to drop the current database. PG throws the
following exception.
{code:java}
test=# drop database
test-# test;
ERROR: cannot drop the currently open database
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-33200:
Fix Version/s: (was: 1.8.4)
> ItemAt Expression validation fail in Table API due to type mismatch
> ---
>
> Key: FLINK-33200
> URL: https://issues.apache.org/jira/browse/FLINK-33200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.18.0, 1.17.1
>Reporter: Zhenqiu Huang
>Priority: Minor
>
> The table schema is defined as below:
> public static final DataType DATA_TYPE = DataTypes.ROW(
> DataTypes.FIELD("id", DataTypes.STRING()),
> DataTypes.FIELD("events",
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
> );
> public static final Schema SCHEMA =
> Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
> inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
> The validation fail as "eventType" is inferred as
> BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
> StringDataTypeInfo. The validation fail at
> case mti: MapTypeInfo[_, _] =>
> if (key.resultType == mti.getKeyTypeInfo) {
> ValidationSuccess
> } else {
> ValidationFailure(
> s"Map entry access needs a valid key of type " +
> s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lincoln lee updated FLINK-33200:
Affects Version/s: (was: 1.18.1)
> ItemAt Expression validation fail in Table API due to type mismatch
> ---
>
> Key: FLINK-33200
> URL: https://issues.apache.org/jira/browse/FLINK-33200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.18.0, 1.17.1
>Reporter: Zhenqiu Huang
>Priority: Minor
> Fix For: 1.8.4
>
>
> The table schema is defined as below:
> public static final DataType DATA_TYPE = DataTypes.ROW(
> DataTypes.FIELD("id", DataTypes.STRING()),
> DataTypes.FIELD("events",
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
> );
> public static final Schema SCHEMA =
> Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
> inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
> The validation fail as "eventType" is inferred as
> BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
> StringDataTypeInfo. The validation fail at
> case mti: MapTypeInfo[_, _] =>
> if (key.resultType == mti.getKeyTypeInfo) {
> ValidationSuccess
> } else {
> ValidationFailure(
> s"Map entry access needs a valid key of type " +
> s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-25593][table-planner] Skip redundant scan while partition table push down contains none-existent partition [flink]
swuferhong commented on PR #23423: URL: https://github.com/apache/flink/pull/23423#issuecomment-1754162270 > FLINK-27898 > @swuferhong Thanks for the pr. Overall, LGTM in code level. But I have one concern about it, in batch mode, it's total right to reduce the empty partition to a single Values node. But in stream mode, is it expected? I mean in optimize phase, no match partitions, but user expect to monitor the match partitions continuously. If reduce it to a single Values node, user will lose the ability. A similar issue it reported in [FLINK-27898](https://issues.apache.org/jira/browse/FLINK-27898). Hi, @luoyuxia . I think it's another issue. The problem is that the partition we push down is static and has already been determined during optimize phase, for streaming mode, it will not work for some cases. However, I think these two issues need consider separately. Currently, this PR is only an optimization under the existing `SupportsPartitionPushDown`. If we want to better support partition push down in streaming mode, I think we need new flip and pr to improve the existing interface. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33206) Verify the existence of hbase table before read/write
[ https://issues.apache.org/jira/browse/FLINK-33206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-33206: --- Labels: pull-request-available stale-blocker (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as a Blocker but is unassigned and neither itself nor its Sub-Tasks have been updated for 1 days. I have gone ahead and marked it "stale-blocker". If this ticket is a Blocker, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Verify the existence of hbase table before read/write > - > > Key: FLINK-33206 > URL: https://issues.apache.org/jira/browse/FLINK-33206 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase >Affects Versions: hbase-3.0.1 >Reporter: tanjialiang >Priority: Blocker > Labels: pull-request-available, stale-blocker > Attachments: image-2023-10-08-16-54-05-917.png > > > Currently, we do not verify the existence of hbase table before read/write, > and the error would make the user confused. > The `HBaseSinkFunction` throws `TableNotFoundException` when do flush. > The `inputFormat` throws not obvious enough. > !image-2023-10-08-16-54-05-917.png! > So i think we should verify the existence of hbase table when call `open` > function. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32714) JDBC: Add dialect for OceanBase database
[
https://issues.apache.org/jira/browse/FLINK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32714:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Major, please
raise the priority and ask a committer to assign you the issue or revive the
public discussion.
> JDBC: Add dialect for OceanBase database
>
>
> Key: FLINK-32714
> URL: https://issues.apache.org/jira/browse/FLINK-32714
> Project: Flink
> Issue Type: New Feature
>Reporter: He Wang
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> OceanBase is a distributed relational database, the community edition of
> OceanBase is open sourced at [https://github.com/oceanbase/oceanbase.]
> The enterprise edition of OceanBase is compatible with MySql and Oracle,
> which means we can reuse almost all the dialect rules.
> The difference from other databases is that we must provide the compatibility
> mode firstly, then the connector can determine which dialect to use, so a
> startup option like 'compatible-mode' is needed.
> A dialect implementation for OceanBase is like below:
> {code:java}
> package org.apache.flink.connector.jdbc.databases.oceanbase;
> import org.apache.flink.connector.jdbc.converter.JdbcRowConverter;
> import org.apache.flink.connector.jdbc.databases.mysql.dialect.MySqlDialect;
> import org.apache.flink.connector.jdbc.databases.oracle.dialect.OracleDialect;
> import org.apache.flink.connector.jdbc.dialect.AbstractDialect;
> import org.apache.flink.table.types.logical.LogicalTypeRoot;
> import org.apache.flink.table.types.logical.RowType;
> import javax.annotation.Nonnull;
> import java.util.Optional;
> import java.util.Set;
> /** JDBC dialect for OceanBase. */
> public class OceanBaseDialect extends AbstractDialect {
> private static final long serialVersionUID = 1L;
> private final AbstractDialect dialect;
> public OceanBaseDialect(@Nonnull String compatibleMode) {
> switch (compatibleMode.toLowerCase()) {
> case "mysql":
> this.dialect = new MySqlDialect();
> break;
> case "oracle":
> this.dialect = new OracleDialect();
> break;
> default:
> throw new IllegalArgumentException(
> "Unsupported compatible mode: " + compatibleMode);
> }
> }
> @Override
> public String dialectName() {
> return "OceanBase";
> }
> @Override
> public Optional defaultDriverName() {
> return Optional.of("com.oceanbase.jdbc.Driver");
> }
> @Override
> public Set supportedTypes() {
> return dialect.supportedTypes();
> }
> @Override
> public JdbcRowConverter getRowConverter(RowType rowType) {
> return dialect.getRowConverter(rowType);
> }
> @Override
> public String getLimitClause(long limit) {
> return dialect.getLimitClause(limit);
> }
> @Override
> public String quoteIdentifier(String identifier) {
> return dialect.quoteIdentifier(identifier);
> }
> @Override
> public Optional getUpsertStatement(
> String tableName, String[] fieldNames, String[] conditionFields) {
> return dialect.getUpsertStatement(tableName, fieldNames,
> conditionFields);
> }
> @Override
> public Optional timestampPrecisionRange() {
> return dialect.timestampPrecisionRange();
> }
> @Override
> public Optional decimalPrecisionRange() {
> return dialect.decimalPrecisionRange();
> }
> @Override
> public String appendDefaultUrlProperties(String url) {
> return dialect.appendDefaultUrlProperties(url);
> }
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32720) Add GENERATE_SERIES support in SQL & Table API
[
https://issues.apache.org/jira/browse/FLINK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-32720:
---
Labels: pull-request-available stale-major (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Add GENERATE_SERIES support in SQL & Table API
> --
>
> Key: FLINK-32720
> URL: https://issues.apache.org/jira/browse/FLINK-32720
> Project: Flink
> Issue Type: Improvement
>Reporter: Hanyu Zheng
>Priority: Major
> Labels: pull-request-available, stale-major
>
> GENERATE_SERIES Function
> Description
> Constructs an array of values between {{start}} and {{{}end{}}}, inclusive.
> Parameters {{start}} and {{end}} can be an {{INT}} or {{{}BIGINT{}}}.
> {{{}step{}}}, if supplied, specifies the step size. The step can be positive
> or negative. If not supplied, {{step}} defaults to {{{}1{}}}. Parameter
> {{step}} must be an {{{}INT{}}}.
> Syntax
> The syntax for the GENERATE_SERIES function is:
> {code:java}
> GENERATE_SERIES(start, end)
> GENERATE_SERIES(start, end, step){code}
>
> Example
> Let's look at some function examples and explore how to use the SPLIT
> function.
> For example:
>
> {code:java}
> SELECT GENERATE_SERISE(1, 5);
> Result: [1,2,3,4,5]
> SELECT GENERATE_SERISE(0, 10, 2);
> Result: [0, 2, 4, 6, 8, 10] {code}
> see also:
> 1.PostgreSQL: PostgreSQL offers a function called {{generate_series}} which
> generates a set of contiguous integers from a start to an end value. An
> optional 'step' parameter is available to specify the increment between each
> integer.
> https://www.postgresql.org/docs/current/functions-srf.html
> 2.ksqlDB: As you mentioned, ksqlDB provides a function called
> {{GENERATE_SERIES}} that generates a series of numbers, starting from a given
> start value, incrementing each time by a step value, until it reaches or
> exceeds a given end value.
> https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/scalar-functions/#generate_series
> 3.BigQuery: BigQuery has a function called {{GENERATE_ARRAY}} that generates
> an array consisting of integers from the start value to the end value, with
> each integer incremented by the step value. You can find more details in the
> https://cloud.google.com/bigquery/docs/reference/standard-sql/array_functions#generate_array
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33207) Scan hbase table will throw error when table is empty
[ https://issues.apache.org/jira/browse/FLINK-33207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-33207: --- Labels: pull-request-available stale-blocker (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as a Blocker but is unassigned and neither itself nor its Sub-Tasks have been updated for 1 days. I have gone ahead and marked it "stale-blocker". If this ticket is a Blocker, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Scan hbase table will throw error when table is empty > - > > Key: FLINK-33207 > URL: https://issues.apache.org/jira/browse/FLINK-33207 > Project: Flink > Issue Type: Bug > Components: Connectors / HBase >Affects Versions: hbase-3.0.1 >Reporter: tanjialiang >Priority: Blocker > Labels: pull-request-available, stale-blocker > > When i scan the empty hbase table, it will throw an error when > createInputSplits, we should return empty split instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-28229][streaming-java] Source API alternatives for StreamExecutionEnvironment#fromCollection() methods [flink]
afedulov commented on PR #22850: URL: https://github.com/apache/flink/pull/22850#issuecomment-1753896916 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests
[
https://issues.apache.org/jira/browse/FLINK-33175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773192#comment-17773192
]
Jing Ge edited comment on FLINK-33175 at 10/9/23 9:03 PM:
--
Thank you all! Appreciate it!
[~mapohl] I just created another umbrella one
https://issues.apache.org/jira/browse/FLINK-33215 at the same time. -May I
convert the ticket you created to be the subtask of mine?- Never mind, I made
my task be blocked by yours.
was (Author: jingge):
Thank you all! Appreciate it!
[~mapohl] I just created another umbrella one
https://issues.apache.org/jira/browse/FLINK-33215 at the same time. May I
convert the ticket you created to be the subtask of mine?
> Nightly builds from S3 are not available for download, breaking all connector
> tests
> ---
>
> Key: FLINK-33175
> URL: https://issues.apache.org/jira/browse/FLINK-33175
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Reporter: Martijn Visser
>Assignee: Jing Ge
>Priority: Blocker
>
> All downloads of Flink binaries fail with:
> {code:java}
> Run wget -q -c
> https://s3.amazonaws.com/flink-nightly/flink-1.18-SNAPSHOT-bin-scala_2.12.tgz
> -O - | tar -xz
> gzip: stdin: unexpected end of file
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> Error: Process completed with exit code 2.
> {code}
> This goes for 1.18, but also 1.17 and 1.16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests
[
https://issues.apache.org/jira/browse/FLINK-33175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773174#comment-17773174
]
Jing Ge edited comment on FLINK-33175 at 10/9/23 9:01 PM:
--
Ververica has changed the files to be publicly accessible. Could anyone of you
trigger any builds to test? Thanks! [~mapohl] [~martijnvisser] [~Sergey
Nuyanzin] [~hlteoh37] [~danny.cranmer]
was (Author: jingge):
Ververica has changed the files to be publicly accessible. Could anyone of you
trigger the any builds to test? Thanks! [~mapohl] [~martijnvisser] [~Sergey
Nuyanzin] [~hlteoh37] [~danny.cranmer]
> Nightly builds from S3 are not available for download, breaking all connector
> tests
> ---
>
> Key: FLINK-33175
> URL: https://issues.apache.org/jira/browse/FLINK-33175
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Reporter: Martijn Visser
>Assignee: Jing Ge
>Priority: Blocker
>
> All downloads of Flink binaries fail with:
> {code:java}
> Run wget -q -c
> https://s3.amazonaws.com/flink-nightly/flink-1.18-SNAPSHOT-bin-scala_2.12.tgz
> -O - | tar -xz
> gzip: stdin: unexpected end of file
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> Error: Process completed with exit code 2.
> {code}
> This goes for 1.18, but also 1.17 and 1.16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
mtfelisb commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-1753824138 > hi @mtfelisb , I read the code and found that it has no implementation for `DynamicTableSink`, hence this es8 sink cannot be used in flink-sql. Would you have any plan to work on this point? Hi, @rinkako! I wasn't planning to do it. To be honest with you I'm not quite familiar with it. However, it could be a significant increment, indeed. I'm open to collaborations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773466#comment-17773466
]
Andriy Onyshchuk edited comment on FLINK-33200 at 10/9/23 7:46 PM:
---
Hello, team. Here is a bit of the summary to this bug
h3. Given
Working on projection over dataset that contains data represented as array of
maps of
string to string `array[map[string, string]]` several issues was identified.
h3. Data Schema
{{DataTypes.ROW(}}
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("events", DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(),
DataTypes.STRING(
{{{}){}}}
h3. Issues
h4. Issue 1. Extraction of map values by index and key.
{{inTable.select(}}
{{ Expressions.$("events").at(1).at("eventType").as("firstEventType")}}
{{).execute().print();}}
results in
{{Exception in thread "main" org.apache.flink.table.api.ValidationException:
Map entry access needs a valid key of type 'StringData', found 'String'.}}
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.validateArguments(PlannerTypeInferenceUtilImpl.java:111)
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.runTypeInference(PlannerTypeInferenceUtilImpl.java:69)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runLegacyTypeInference(ResolveCallByArgumentsRule.java:284)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$2(ResolveCallByArgumentsRule.java:164)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:164)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:147)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:92)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:93)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.lambda$concatenateRules$2(ExpressionResolver.java:247)
at java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.resolve(ExpressionResolver.java:210)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.projectInternal(OperationTreeBuilder.java:199)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.project(OperationTreeBuilder.java:174)
at org.apache.flink.table.api.internal.TableImpl.select(TableImpl.java:142)
at IssueDemo.main(IssueDemo.java:73)}}
h4. Issue 2. Extraction of entire map by index.
{{inTable.select(}}
{{ Expressions.$("events").at(1).as("firstEvent")}}
{{).execute().print();}}
results in
{{Query schema: [firstEvent: MAP]}}
{{Sink schema: [firstEvent: MAP]}}
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.createSchemaMismatchException(DynamicSinkUtils.java:1005)
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:349)
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:307)
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.convertSinkToRel(DynamicSinkUtils.java:268)
at
[jira] [Comment Edited] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773466#comment-17773466
]
Andriy Onyshchuk edited comment on FLINK-33200 at 10/9/23 7:45 PM:
---
Hello, team. Here is a bit of the summary to this bug
h3. Given
Working on projection over dataset that contains data represented as array of
maps of
string to string `array[map[string, string]]` several issues was identified.
h3. Data Schema
{{DataTypes.ROW(}}
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("events", DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(),
DataTypes.STRING(
{{{}){}}}
h3. Issues
h4. Issue 1. Extraction of map values by index and key.
{{inTable.select(}}
{{ Expressions.$("events").at(1).at("eventType").as("firstEventType")}}
{{).execute().print();}}
results in
{{Exception in thread "main" org.apache.flink.table.api.ValidationException:
Map entry access needs a valid key of type 'StringData', found 'String'.}}
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.validateArguments(PlannerTypeInferenceUtilImpl.java:111)
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.runTypeInference(PlannerTypeInferenceUtilImpl.java:69)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runLegacyTypeInference(ResolveCallByArgumentsRule.java:284)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$2(ResolveCallByArgumentsRule.java:164)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:164)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:147)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:92)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:93)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.lambda$concatenateRules$2(ExpressionResolver.java:247)
at java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.resolve(ExpressionResolver.java:210)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.projectInternal(OperationTreeBuilder.java:199)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.project(OperationTreeBuilder.java:174)
at org.apache.flink.table.api.internal.TableImpl.select(TableImpl.java:142)
at IssueDemo.main(IssueDemo.java:73)}}
h4. Issue 2. Extraction of entire map by index.
{{inTable.select(}}
{{ Expressions.$("events").at(1).as("firstEvent")}}
{{).execute().print();}}
results in
{{Query schema: [firstEvent: MAP]}}
{{Sink schema: [firstEvent: MAP]}}
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.createSchemaMismatchException(DynamicSinkUtils.java:1005)
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:349)
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:307)
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.convertSinkToRel(DynamicSinkUtils.java:268)
at
[jira] [Comment Edited] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773466#comment-17773466
]
Andriy Onyshchuk edited comment on FLINK-33200 at 10/9/23 7:43 PM:
---
Hello, team. Here is a bit of the summary to this bug
h3. Given
Working on projection over dataset that contains data represented as array of
maps of
string to string `array[map[string, string]]` several issues was identified.
h3. Data Schema
{{DataTypes.ROW(}}
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("events", DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(),
DataTypes.STRING(
{{{}){}}}
h3. Issues
h4. Issue 1. Extraction of map values by index and key.
{{inTable.select(}}
{{ Expressions.$("events").at(1).at("eventType").as("firstEventType")}}
{{).execute().print();}}
results in
{{Exception in thread "main" org.apache.flink.table.api.ValidationException:
Map entry access needs a valid key of type 'StringData', found 'String'.}}
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.validateArguments(PlannerTypeInferenceUtilImpl.java:111)
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.runTypeInference(PlannerTypeInferenceUtilImpl.java:69)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runLegacyTypeInference(ResolveCallByArgumentsRule.java:284)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$2(ResolveCallByArgumentsRule.java:164)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:164)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:147)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:92)
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:93)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.lambda$concatenateRules$2(ExpressionResolver.java:247)
at java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.resolve(ExpressionResolver.java:210)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.projectInternal(OperationTreeBuilder.java:199)
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.project(OperationTreeBuilder.java:174)
at org.apache.flink.table.api.internal.TableImpl.select(TableImpl.java:142)
at IssueDemo.main(IssueDemo.java:73)}}
h4. Issue 2. Extraction of entire map by index.
{{inTable.select(}}
{{ Expressions.$("events").at(1).as("firstEvent")}}
{{).execute().print();}}
results in
{{{}Query schema: [firstEvent: MAP]{}}}{{{}Sink schema:
[firstEvent: MAP]{}}}{\{ at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.createSchemaMismatchException(DynamicSinkUtils.java:1005)}}\{{
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:349)}}\{{
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:307)}}\{{
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.convertSinkToRel(DynamicSinkUtils.java:268)}}\{{
at
[jira] [Comment Edited] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773466#comment-17773466
]
Andriy Onyshchuk edited comment on FLINK-33200 at 10/9/23 7:40 PM:
---
Hello, team. Here is a bit of the summary to this bug
h3. Given
Working on projection over dataset that contains data represented as array of
maps of
string to string `array[map[string, string]]` several issues was identified.
h3. Data Schema
{{DataTypes.ROW(}}
DataTypes.FIELD("id", DataTypes.STRING()),
DataTypes.FIELD("events", DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(),
DataTypes.STRING(
{{{}){}}}
h3. Issues
h4. Issue 1. Extraction of map values by index and key.
{{inTable.select(}}
{{ Expressions.$("events").at(1).at("eventType").as("firstEventType")}}
{{).execute().print();}}
results in
{{{}Exception in thread "main" org.apache.flink.table.api.ValidationException:
Map entry access needs a valid key of type 'StringData', found 'String'.{}}}{\{
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.validateArguments(PlannerTypeInferenceUtilImpl.java:111)}}\{{
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.runTypeInference(PlannerTypeInferenceUtilImpl.java:69)}}\{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runLegacyTypeInference(ResolveCallByArgumentsRule.java:284)}}\{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$2(ResolveCallByArgumentsRule.java:164)}}\{{
at java.base/java.util.Optional.orElseGet(Optional.java:369)}}\{{ at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:164)}}\{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)}}\{{
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)}}\{{
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)}}\{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:147)}}\{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)}}\{{
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)}}\{{
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)}}\{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:92)}}\{{
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)}}\{{
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)}}\{{
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)}}\{{
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)}}\{{
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)}}\{{
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)}}\{{
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)}}\{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:93)}}\{{
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.lambda$concatenateRules$2(ExpressionResolver.java:247)}}\{{
at
java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)}}\{{
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.resolve(ExpressionResolver.java:210)}}\{{
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.projectInternal(OperationTreeBuilder.java:199)}}\{{
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.project(OperationTreeBuilder.java:174)}}\{{
at
org.apache.flink.table.api.internal.TableImpl.select(TableImpl.java:142)}}\{{
at IssueDemo.main(IssueDemo.java:73)}}
h4. Issue 2. Extraction of entire map by index.
{{inTable.select(}}
{{ Expressions.$("events").at(1).as("firstEvent")}}
{{).execute().print();}}
results in
{{{}Query schema: [firstEvent: MAP]{}}}{{{}Sink schema:
[firstEvent: MAP]{}}}{\{ at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.createSchemaMismatchException(DynamicSinkUtils.java:1005)}}\{{
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:349)}}\{{
at
[jira] [Commented] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773466#comment-17773466
]
Andriy Onyshchuk commented on FLINK-33200:
--
Hello, team. Here is a bit of the summary to this bug
h3. Given
Working on projection over dataset that contains data represented as array of
maps of
string to string `array[map[string, string]]` several issues was identified.
h3. Data Schema
{{DataTypes.ROW(}}
{{ DataTypes.FIELD("id", DataTypes.STRING()),}}
{{ DataTypes.FIELD("events",
DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(}}
{{{}){}}}
h3. Issues
h4. Issue 1. Extraction of map values by index and key.
{{inTable.select(}}
{{ Expressions.$("events").at(1).at("eventType").as("firstEventType")}}
{{).execute().print();}}
results in
{{Exception in thread "main" org.apache.flink.table.api.ValidationException:
Map entry access needs a valid key of type 'StringData', found 'String'.}}{{ at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.validateArguments(PlannerTypeInferenceUtilImpl.java:111)}}{{
at
org.apache.flink.table.planner.expressions.PlannerTypeInferenceUtilImpl.runTypeInference(PlannerTypeInferenceUtilImpl.java:69)}}{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.runLegacyTypeInference(ResolveCallByArgumentsRule.java:284)}}{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.lambda$visit$2(ResolveCallByArgumentsRule.java:164)}}{{
at java.base/java.util.Optional.orElseGet(Optional.java:369)}}{{ at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:164)}}{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)}}{{
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)}}{{
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)}}{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:147)}}{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule$ResolvingCallVisitor.visit(ResolveCallByArgumentsRule.java:98)}}{{
at
org.apache.flink.table.expressions.ApiExpressionVisitor.visit(ApiExpressionVisitor.java:37)}}{{
at
org.apache.flink.table.expressions.UnresolvedCallExpression.accept(UnresolvedCallExpression.java:97)}}{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.lambda$apply$0(ResolveCallByArgumentsRule.java:92)}}{{
at
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271)}}{{
at
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)}}{{
at
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)}}{{
at
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)}}{{
at
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)}}{{
at
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)}}{{
at
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)}}{{
at
org.apache.flink.table.expressions.resolver.rules.ResolveCallByArgumentsRule.apply(ResolveCallByArgumentsRule.java:93)}}{{
at
org.apache.flink.table.expressions.resolver.ExpressionResolver.lambda$concatenateRules$2(ExpressionResolver.java:247)}}{{
at
java.base/java.util.function.Function.lambda$andThen$1(Function.java:88)}}{{ at
org.apache.flink.table.expressions.resolver.ExpressionResolver.resolve(ExpressionResolver.java:210)}}{{
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.projectInternal(OperationTreeBuilder.java:199)}}{{
at
org.apache.flink.table.operations.utils.OperationTreeBuilder.project(OperationTreeBuilder.java:174)}}{{
at
org.apache.flink.table.api.internal.TableImpl.select(TableImpl.java:142)}}{{ at
IssueDemo.main(IssueDemo.java:73)}}
h4. Issue 2. Extraction of entire map by index.
{{inTable.select(}}
{{ Expressions.$("events").at(1).as("firstEvent")}}
{{).execute().print();}}
results in
{{{}Query schema: [firstEvent: MAP]{}}}{{{}Sink schema:
[firstEvent: MAP]{}}}{{ at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.createSchemaMismatchException(DynamicSinkUtils.java:1005)}}{{
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:349)}}{{
at
org.apache.flink.table.planner.connectors.DynamicSinkUtils.validateSchemaAndApplyImplicitCast(DynamicSinkUtils.java:307)}}{{
at
Re: [PR] [FLINK-25823] Remove Mesos from flink Architecture documentation. [flink-web]
Samrat002 commented on PR #682: URL: https://github.com/apache/flink-web/pull/682#issuecomment-1753451191 @MartijnVisser made changes for chinese doc as well . please review whenever time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33225) Python API incorrectly passes `JVM_ARGS` as single argument
[
https://issues.apache.org/jira/browse/FLINK-33225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Deepyaman Datta updated FLINK-33225:
Description:
In the same vein as https://issues.apache.org/jira/browse/FLINK-31915,
`JVM_ARGS` need to be passed as an array. For example, the current behavior of
export `JVM_ARGS='-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M'`
is:
{{> raise RuntimeError(}}
{{ "Java gateway process exited before sending its port
number.\nStderr:\n"}}
{{ + stderr_info}}
{{ )}}
{{E RuntimeError: Java gateway process exited before sending its
port number.}}
{{E Stderr:}}
{{E Improperly specified VM option 'CompressedClassSpaceSize=100M
-XX:MaxMetaspaceSize=200M'}}
{{E Error: Could not create the Java Virtual Machine.}}
{{E Error: A fatal exception has occurred. Program will exit.}}
was:
In the same vein as https://issues.apache.org/jira/browse/FLINK-31915,
`JVM_ARGS` need to be passed as an array. For example, the current behavior of
`export JVM_ARGS='-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M'`
is:
```
> raise RuntimeError(
"Java gateway process exited before sending its port
number.\nStderr:\n"
+ stderr_info
)
E RuntimeError: Java gateway process exited before sending its
port number.
E Stderr:
E Improperly specified VM option 'CompressedClassSpaceSize=100M
-XX:MaxMetaspaceSize=200M'
E Error: Could not create the Java Virtual Machine.
E Error: A fatal exception has occurred. Program will exit.
> Python API incorrectly passes `JVM_ARGS` as single argument
> ---
>
> Key: FLINK-33225
> URL: https://issues.apache.org/jira/browse/FLINK-33225
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.18.0, 1.17.1, 1.18.1
>Reporter: Deepyaman Datta
>Priority: Major
> Labels: github-pullrequest
>
> In the same vein as https://issues.apache.org/jira/browse/FLINK-31915,
> `JVM_ARGS` need to be passed as an array. For example, the current behavior
> of export `JVM_ARGS='-XX:CompressedClassSpaceSize=100M
> -XX:MaxMetaspaceSize=200M'` is:
> {{> raise RuntimeError(}}
> {{ "Java gateway process exited before sending its port
> number.\nStderr:\n"}}
> {{ + stderr_info}}
> {{ )}}
> {{E RuntimeError: Java gateway process exited before sending
> its port number.}}
> {{E Stderr:}}
> {{E Improperly specified VM option
> 'CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M'}}
> {{E Error: Could not create the Java Virtual Machine.}}
> {{E Error: A fatal exception has occurred. Program will exit.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33225) Python API incorrectly passes `JVM_ARGS` as single argument
[ https://issues.apache.org/jira/browse/FLINK-33225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepyaman Datta updated FLINK-33225: Description: In the same vein as https://issues.apache.org/jira/browse/FLINK-31915, `JVM_ARGS` need to be passed as an array. For example, the current behavior of `export JVM_ARGS='-XX:CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M'` is: ``` > raise RuntimeError( "Java gateway process exited before sending its port number.\nStderr:\n" + stderr_info ) E RuntimeError: Java gateway process exited before sending its port number. E Stderr: E Improperly specified VM option 'CompressedClassSpaceSize=100M -XX:MaxMetaspaceSize=200M' E Error: Could not create the Java Virtual Machine. E Error: A fatal exception has occurred. Program will exit. > Python API incorrectly passes `JVM_ARGS` as single argument > --- > > Key: FLINK-33225 > URL: https://issues.apache.org/jira/browse/FLINK-33225 > Project: Flink > Issue Type: Bug >Affects Versions: 1.18.0, 1.17.1, 1.18.1 >Reporter: Deepyaman Datta >Priority: Major > Labels: github-pullrequest > > In the same vein as https://issues.apache.org/jira/browse/FLINK-31915, > `JVM_ARGS` need to be passed as an array. For example, the current behavior > of `export JVM_ARGS='-XX:CompressedClassSpaceSize=100M > -XX:MaxMetaspaceSize=200M'` is: > ``` > > raise RuntimeError( > "Java gateway process exited before sending its port > number.\nStderr:\n" > + stderr_info > ) > E RuntimeError: Java gateway process exited before sending its > port number. > E Stderr: > E Improperly specified VM option 'CompressedClassSpaceSize=100M > -XX:MaxMetaspaceSize=200M' > E Error: Could not create the Java Virtual Machine. > E Error: A fatal exception has occurred. Program will exit. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33225) Python API incorrectly passes `JVM_ARGS` as single argument
[ https://issues.apache.org/jira/browse/FLINK-33225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepyaman Datta updated FLINK-33225: Affects Version/s: 1.17.1 1.18.0 1.18.1 > Python API incorrectly passes `JVM_ARGS` as single argument > --- > > Key: FLINK-33225 > URL: https://issues.apache.org/jira/browse/FLINK-33225 > Project: Flink > Issue Type: Bug >Affects Versions: 1.18.0, 1.17.1, 1.18.1 >Reporter: Deepyaman Datta >Priority: Major > Labels: github-pullrequest > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33225) Python API incorrectly passes `JVM_ARGS` as single argument
[ https://issues.apache.org/jira/browse/FLINK-33225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepyaman Datta updated FLINK-33225: Labels: github-pullrequest (was: ) > Python API incorrectly passes `JVM_ARGS` as single argument > --- > > Key: FLINK-33225 > URL: https://issues.apache.org/jira/browse/FLINK-33225 > Project: Flink > Issue Type: Bug >Reporter: Deepyaman Datta >Priority: Major > Labels: github-pullrequest > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33225) Python API incorrectly passes `JVM_ARGS` as single argument
Deepyaman Datta created FLINK-33225: --- Summary: Python API incorrectly passes `JVM_ARGS` as single argument Key: FLINK-33225 URL: https://issues.apache.org/jira/browse/FLINK-33225 Project: Flink Issue Type: Bug Reporter: Deepyaman Datta -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-XXXXX][python] Parse `JVM_ARGS` as an array [flink]
deepyaman commented on PR #23500: URL: https://github.com/apache/flink/pull/23500#issuecomment-1753392855 FYI @zentol since this is very similar to your change in #22476. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-XXXXX][python] Parse `JVM_ARGS` as an array [flink]
flinkbot commented on PR #23500: URL: https://github.com/apache/flink/pull/23500#issuecomment-1753387268 ## CI report: * 96c57238233a7eb8dd63a25b14b6188375888c16 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-XXXXX][python] Parse `JVM_ARGS` as an array [flink]
deepyaman opened a new pull request, #23500: URL: https://github.com/apache/flink/pull/23500 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhenqiu Huang updated FLINK-33200:
--
Affects Version/s: 1.17.1
1.18.0
1.18.1
> ItemAt Expression validation fail in Table API due to type mismatch
> ---
>
> Key: FLINK-33200
> URL: https://issues.apache.org/jira/browse/FLINK-33200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.18.0, 1.17.1, 1.18.1
>Reporter: Zhenqiu Huang
>Priority: Minor
> Fix For: 1.8.4
>
>
> The table schema is defined as below:
> public static final DataType DATA_TYPE = DataTypes.ROW(
> DataTypes.FIELD("id", DataTypes.STRING()),
> DataTypes.FIELD("events",
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
> );
> public static final Schema SCHEMA =
> Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
> inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
> The validation fail as "eventType" is inferred as
> BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
> StringDataTypeInfo. The validation fail at
> case mti: MapTypeInfo[_, _] =>
> if (key.resultType == mti.getKeyTypeInfo) {
> ValidationSuccess
> } else {
> ValidationFailure(
> s"Map entry access needs a valid key of type " +
> s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773373#comment-17773373
]
Zhenqiu Huang commented on FLINK-33200:
---
We tried both 1.17.1 and 1.18-SNAPSHOT
> ItemAt Expression validation fail in Table API due to type mismatch
> ---
>
> Key: FLINK-33200
> URL: https://issues.apache.org/jira/browse/FLINK-33200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Zhenqiu Huang
>Priority: Minor
> Fix For: 1.8.4
>
>
> The table schema is defined as below:
> public static final DataType DATA_TYPE = DataTypes.ROW(
> DataTypes.FIELD("id", DataTypes.STRING()),
> DataTypes.FIELD("events",
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
> );
> public static final Schema SCHEMA =
> Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
> inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
> The validation fail as "eventType" is inferred as
> BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
> StringDataTypeInfo. The validation fail at
> case mti: MapTypeInfo[_, _] =>
> if (key.resultType == mti.getKeyTypeInfo) {
> ValidationSuccess
> } else {
> ValidationFailure(
> s"Map entry access needs a valid key of type " +
> s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests
[
https://issues.apache.org/jira/browse/FLINK-33175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773372#comment-17773372
]
Jing Ge commented on FLINK-33175:
-
[~hong] Thanks for the effort and feedback!
> Nightly builds from S3 are not available for download, breaking all connector
> tests
> ---
>
> Key: FLINK-33175
> URL: https://issues.apache.org/jira/browse/FLINK-33175
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Reporter: Martijn Visser
>Assignee: Jing Ge
>Priority: Blocker
>
> All downloads of Flink binaries fail with:
> {code:java}
> Run wget -q -c
> https://s3.amazonaws.com/flink-nightly/flink-1.18-SNAPSHOT-bin-scala_2.12.tgz
> -O - | tar -xz
> gzip: stdin: unexpected end of file
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> Error: Process completed with exit code 2.
> {code}
> This goes for 1.18, but also 1.17 and 1.16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-15462][connectors] Add Trino dialect [flink-connector-jdbc]
eskabetxe commented on PR #3: URL: https://github.com/apache/flink-connector-jdbc/pull/3#issuecomment-1753222806 @snuyanzin rebased and tests passing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33182) Allow metadata columns in NduAnalyzer with ChangelogNormalize
[ https://issues.apache.org/jira/browse/FLINK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773370#comment-17773370 ] Timo Walther commented on FLINK-33182: -- Great that we are on the same page. Yes I meant the `StreamNonDeterministicUpdatePlanVisitor`. Do you want to take this ticket? I guess you are the expert in this class. > Allow metadata columns in NduAnalyzer with ChangelogNormalize > - > > Key: FLINK-33182 > URL: https://issues.apache.org/jira/browse/FLINK-33182 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > Currently, the NduAnalyzer is very strict about metadata columns in updating > sources. However, for upsert sources (like Kafka) that contain an incomplete > changelog, the planner always adds a ChangelogNormalize node. > ChangelogNormalize will make sure that metadata columns can be considered > deterministic. So the NduAnalyzer should be satisfied in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33182) Allow metadata columns in NduAnalyzer with ChangelogNormalize
[ https://issues.apache.org/jira/browse/FLINK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773367#comment-17773367 ] lincoln lee commented on FLINK-33182: - [~twalthr] You're right, I checked the current `DeduplicateFunctionHelper#processLastRowOnChangelog` processing, it does avoid the non-determinism of the -D message, so I agree it's safe in this case with the guarantee that metadata columns are not allowed as primary key. By the way, when you mention NduAnalyzer here, do you actually mean StreamNonDeterministicUpdatePlanVisitor (because NDUAnalyzer relies on it, and we should actually modify the visitor)? > Allow metadata columns in NduAnalyzer with ChangelogNormalize > - > > Key: FLINK-33182 > URL: https://issues.apache.org/jira/browse/FLINK-33182 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > Currently, the NduAnalyzer is very strict about metadata columns in updating > sources. However, for upsert sources (like Kafka) that contain an incomplete > changelog, the planner always adds a ChangelogNormalize node. > ChangelogNormalize will make sure that metadata columns can be considered > deterministic. So the NduAnalyzer should be satisfied in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33200) ItemAt Expression validation fail in Table API due to type mismatch
[
https://issues.apache.org/jira/browse/FLINK-33200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773361#comment-17773361
]
lincoln lee commented on FLINK-33200:
-
[~ZhenqiuHuang] What flink version are you using?
> ItemAt Expression validation fail in Table API due to type mismatch
> ---
>
> Key: FLINK-33200
> URL: https://issues.apache.org/jira/browse/FLINK-33200
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Zhenqiu Huang
>Priority: Minor
> Fix For: 1.8.4
>
>
> The table schema is defined as below:
> public static final DataType DATA_TYPE = DataTypes.ROW(
> DataTypes.FIELD("id", DataTypes.STRING()),
> DataTypes.FIELD("events",
> DataTypes.ARRAY(DataTypes.MAP(DataTypes.STRING(), DataTypes.STRING(
> );
> public static final Schema SCHEMA =
> Schema.newBuilder().fromRowDataType(DATA_TYPE).build();
> inTable.select(Expressions.$("events").at(1).at("eventType").as("firstEventType")
> The validation fail as "eventType" is inferred as
> BasicTypeInfo.STRING_TYPE_INFO, the table key internally is a
> StringDataTypeInfo. The validation fail at
> case mti: MapTypeInfo[_, _] =>
> if (key.resultType == mti.getKeyTypeInfo) {
> ValidationSuccess
> } else {
> ValidationFailure(
> s"Map entry access needs a valid key of type " +
> s"'${mti.getKeyTypeInfo}', found '${key.resultType}'.")
> }
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33223] MATCH_RECOGNIZE AFTER MATCH clause can not be deserialised from a compiled plan [flink]
flinkbot commented on PR #23499: URL: https://github.com/apache/flink/pull/23499#issuecomment-1753158834 ## CI report: * 219eaaca4599228b9aa1c46c8555f9fbf4ef99ad UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33224) Flink Batch Job does not complete when using Kafka topics with empty partitions
Kanthi Vaidya created FLINK-33224: - Summary: Flink Batch Job does not complete when using Kafka topics with empty partitions Key: FLINK-33224 URL: https://issues.apache.org/jira/browse/FLINK-33224 Project: Flink Issue Type: Bug Components: Connectors / Kafka Affects Versions: 1.14.6 Reporter: Kanthi Vaidya Kafka Source Connector no longer throws an exception when running a flink job in batch mode when using Kafka topics with empty partitions. However, flink does not emit the final watermark of Long.MAX_VALUE even if all the messages have been read. Note that we are using bounded sources, since that is a requirement for batch mode. It appears that the Kafka consumer continues to run even though the source is bounded. Same is the case with an empty topic. We do have optional sources, where data need not be present all the time. However if we mark the source as bounded, the Kafka connector should recognize empty partitions and mark it as completed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33223) MATCH_RECOGNIZE AFTER MATCH clause can not be deserialised from a compiled plan
[
https://issues.apache.org/jira/browse/FLINK-33223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33223:
---
Labels: pull-request-available (was: )
> MATCH_RECOGNIZE AFTER MATCH clause can not be deserialised from a compiled
> plan
> ---
>
> Key: FLINK-33223
> URL: https://issues.apache.org/jira/browse/FLINK-33223
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Dawid Wysakowicz
>Assignee: Dawid Wysakowicz
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> {code}
> String sql =
> "insert into MySink"
> + " SELECT * FROM\n"
> + " MyTable\n"
> + " MATCH_RECOGNIZE(\n"
> + " PARTITION BY vehicle_id\n"
> + " ORDER BY `rowtime`\n"
> + " MEASURES \n"
> + " FIRST(A.`rowtime`) as startTime,\n"
> + " LAST(A.`rowtime`) as endTime,\n"
> + " FIRST(A.engine_temperature) as
> Initial_Temp,\n"
> + " LAST(A.engine_temperature) as Final_Temp\n"
> + " ONE ROW PER MATCH\n"
> + " AFTER MATCH SKIP TO FIRST B\n"
> + " PATTERN (A+ B)\n"
> + " DEFINE\n"
> + " A as LAST(A.engine_temperature,1) is NULL
> OR A.engine_temperature > LAST(A.engine_temperature,1),\n"
> + " B as B.engine_temperature <
> LAST(A.engine_temperature)\n"
> + " )MR;";
> util.verifyJsonPlan(String.format(sql, afterClause));
> {code}
> fails with:
> {code}
> Could not resolve internal system function '$SKIP TO LAST$1'. This is a bug,
> please file an issue. (through reference chain:
> org.apache.flink.table.planner.plan.nodes.exec.serde.JsonPlanGraph["nodes"]->java.util.ArrayList[3]->org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecMatch["matchSpec"]->org.apache.flink.table.planner.plan.nodes.exec.spec.MatchSpec["after"])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-33223] MATCH_RECOGNIZE AFTER MATCH clause can not be deserialised from a compiled plan [flink]
dawidwys opened a new pull request, #23499: URL: https://github.com/apache/flink/pull/23499 ## What is the purpose of the change Fixes deserialization issues with `AFTER MATCH` clause in `MATCH_RECOGNIZE` ## Brief change log * added `SKIP_TO_LAST` and `SKIP_TO_FIRST` to Flink operator table ## Verifying this change Added tests in `org.apache.flink.table.planner.plan.nodes.exec.stream.MatchRecognizeJsonPlanTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33179] Throw exception when serialising or deserialising ExecNode with invalid type [flink]
dawidwys commented on code in PR #23488:
URL: https://github.com/apache/flink/pull/23488#discussion_r1350384802
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNodeContext.java:
##
@@ -167,12 +170,19 @@ public ExecNodeContext withId(int id) {
*/
@JsonValue
public String getTypeAsString() {
+if (name == null || version == null) {
+throw new TableException(
+String.format(
+"Can not serialize ExecNode with id: %d. Missing
type, this is a bug,"
++ " please file a ticket.",
Review Comment:
The current phrasing is in line with other similar places. I would also not
use a proprietary system name in the code. It's not guaranteed Apache and Flink
in particular continues using JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33179] Throw exception when serialising or deserialising ExecNode with invalid type [flink]
dawidwys commented on code in PR #23488:
URL: https://github.com/apache/flink/pull/23488#discussion_r1350384802
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNodeContext.java:
##
@@ -167,12 +170,19 @@ public ExecNodeContext withId(int id) {
*/
@JsonValue
public String getTypeAsString() {
+if (name == null || version == null) {
+throw new TableException(
+String.format(
+"Can not serialize ExecNode with id: %d. Missing
type, this is a bug,"
++ " please file a ticket.",
Review Comment:
The current phrasing is in line with other similar places. I would also not
use a proprietary system name in the code. It's not said Apache and Flink in
particular continues using JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33179] Throw exception when serialising or deserialising ExecNode with invalid type [flink]
dawidwys commented on code in PR #23488:
URL: https://github.com/apache/flink/pull/23488#discussion_r1350383216
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/UnsupportedNodesInPlanTest.java:
##
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec;
+
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.PlanReference;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.planner.utils.TableTestBase;
+
+import org.junit.Test;
+
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+/** Tests for deserialising invalid {@link
org.apache.flink.table.api.CompiledPlan}. */
+public class UnsupportedNodesInPlanTest extends TableTestBase {
+
+@Test
+public void testInvalidType() {
+final TableEnvironment tEnv =
+TableEnvironment.create(EnvironmentSettings.inStreamingMode());
+assertThatThrownBy(
+() ->
+tEnv.loadPlan(
+PlanReference.fromResource(
+
"/jsonplan/testInvalidTypeJsonPlan.json")))
+.hasRootCauseMessage(
+"Unsupported exec node type: 'null_null'.");
Review Comment:
It is not really possible to achieve this. Please see the discussion:
https://github.com/apache/flink/pull/23488#discussion_r1345505603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33179] Throw exception when serialising or deserialising ExecNode with invalid type [flink]
dawidwys commented on code in PR #23488:
URL: https://github.com/apache/flink/pull/23488#discussion_r1350382539
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/ExecNodeContext.java:
##
@@ -105,6 +105,9 @@ private ExecNodeContext(@Nullable Integer id, String name,
Integer version) {
public ExecNodeContext(String value) {
this.id = null;
String[] split = value.split("_");
+if ("null".equals(split[0]) || "null".equals(split[1])) {
Review Comment:
Please see the discussion here:
https://github.com/apache/flink/pull/23488#discussion_r1345505603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-33223) MATCH_RECOGNIZE AFTER MATCH clause can not be deserialised from a compiled plan
Dawid Wysakowicz created FLINK-33223:
Summary: MATCH_RECOGNIZE AFTER MATCH clause can not be
deserialised from a compiled plan
Key: FLINK-33223
URL: https://issues.apache.org/jira/browse/FLINK-33223
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
Fix For: 1.19.0
{code}
String sql =
"insert into MySink"
+ " SELECT * FROM\n"
+ " MyTable\n"
+ " MATCH_RECOGNIZE(\n"
+ " PARTITION BY vehicle_id\n"
+ " ORDER BY `rowtime`\n"
+ " MEASURES \n"
+ " FIRST(A.`rowtime`) as startTime,\n"
+ " LAST(A.`rowtime`) as endTime,\n"
+ " FIRST(A.engine_temperature) as
Initial_Temp,\n"
+ " LAST(A.engine_temperature) as Final_Temp\n"
+ " ONE ROW PER MATCH\n"
+ " AFTER MATCH SKIP TO FIRST B\n"
+ " PATTERN (A+ B)\n"
+ " DEFINE\n"
+ " A as LAST(A.engine_temperature,1) is NULL OR
A.engine_temperature > LAST(A.engine_temperature,1),\n"
+ " B as B.engine_temperature <
LAST(A.engine_temperature)\n"
+ " )MR;";
util.verifyJsonPlan(String.format(sql, afterClause));
{code}
fails with:
{code}
Could not resolve internal system function '$SKIP TO LAST$1'. This is a bug,
please file an issue. (through reference chain:
org.apache.flink.table.planner.plan.nodes.exec.serde.JsonPlanGraph["nodes"]->java.util.ArrayList[3]->org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecMatch["matchSpec"]->org.apache.flink.table.planner.plan.nodes.exec.spec.MatchSpec["after"])
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33068) Handle DNS errors in AWS connectors as recoverable
[ https://issues.apache.org/jira/browse/FLINK-33068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773329#comment-17773329 ] Danny Cranmer commented on FLINK-33068: --- [~elphastori] done, thanks > Handle DNS errors in AWS connectors as recoverable > -- > > Key: FLINK-33068 > URL: https://issues.apache.org/jira/browse/FLINK-33068 > Project: Flink > Issue Type: Improvement > Components: Connectors / AWS >Affects Versions: 1.15.4, 1.16.2, 1.17.1 >Reporter: Elphas Toringepi >Assignee: Elphas Toringepi >Priority: Major > Fix For: 1.15.4, 1.16.2, 1.17.1 > > > AWS connectors are not treating DNS errors as recoverable using exponential > back-off resulting in Flink job restarts on transient errors and small > timeouts. > > DNS errors typically manifest as UnknownHostException which is not handled as > a recoverable exception and potentially not retried. > > Here is an [example of an > exception|https://github.com/apache/flink-connector-aws/blob/21c64ce38d3a6fd4fd9b4abda416c2de1a926780/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java#L459] > that is recoverable with exponential-backoff. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33068) Handle DNS errors in AWS connectors as recoverable
[ https://issues.apache.org/jira/browse/FLINK-33068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Cranmer reassigned FLINK-33068: - Assignee: Elphas Toringepi > Handle DNS errors in AWS connectors as recoverable > -- > > Key: FLINK-33068 > URL: https://issues.apache.org/jira/browse/FLINK-33068 > Project: Flink > Issue Type: Improvement > Components: Connectors / AWS >Affects Versions: 1.15.4, 1.16.2, 1.17.1 >Reporter: Elphas Toringepi >Assignee: Elphas Toringepi >Priority: Major > Fix For: 1.15.4, 1.16.2, 1.17.1 > > > AWS connectors are not treating DNS errors as recoverable using exponential > back-off resulting in Flink job restarts on transient errors and small > timeouts. > > DNS errors typically manifest as UnknownHostException which is not handled as > a recoverable exception and potentially not retried. > > Here is an [example of an > exception|https://github.com/apache/flink-connector-aws/blob/21c64ce38d3a6fd4fd9b4abda416c2de1a926780/flink-connector-aws/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java#L459] > that is recoverable with exponential-backoff. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33222) Operator rollback app when it should not
Nicolas Fraison created FLINK-33222: --- Summary: Operator rollback app when it should not Key: FLINK-33222 URL: https://issues.apache.org/jira/browse/FLINK-33222 Project: Flink Issue Type: Bug Components: Kubernetes Operator Environment: Flink operator 1.6 - Flink 1.17.1 Reporter: Nicolas Fraison The operator can decide to rollback when an update of the job spec is performed on savepointTriggerNonce or initialSavepointPath if the app has been deployed since more than KubernetesOperatorConfigOptions.DEPLOYMENT_READINESS_TIMEOUT. This is due to the objectmeta generation being [updated|https://github.com/apache/flink-kubernetes-operator/blob/release-1.6/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java#L169] when changing those spec and leading to the lastReconcileSpec not being aligned with the stableReconcileSpec while those spec are well ignored when checking for upgrade diff Looking at the main branch we should still face the same issue as the same [update|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/AbstractFlinkResourceReconciler.java#L169] is performed at the end of the reconcile loop -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33104) Nightly run for Flink Kafka connector fails due to architecture tests failing
[
https://issues.apache.org/jira/browse/FLINK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-33104.
--
Resolution: Duplicate
> Nightly run for Flink Kafka connector fails due to architecture tests failing
> -
>
> Key: FLINK-33104
> URL: https://issues.apache.org/jira/browse/FLINK-33104
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: kafka-3.1.0
>Reporter: Martijn Visser
>Priority: Blocker
>
> {code:java}
> 2023-09-17T00:29:07.1675694Z [WARNING] Tests run: 18, Failures: 0, Errors: 0,
> Skipped: 9, Time elapsed: 308.532 s - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerMigrationTest
> 2023-09-17T00:29:07.5171608Z [INFO]
> 2023-09-17T00:29:07.5172360Z [INFO] Results:
> 2023-09-17T00:29:07.5172773Z [INFO]
> 2023-09-17T00:29:07.5173139Z [ERROR] Failures:
> 2023-09-17T00:29:07.5174181Z [ERROR] Architecture Violation [Priority:
> MEDIUM] - Rule 'ITCASE tests should use a MiniCluster resource or extension'
> was violated (13 times):
> 2023-09-17T00:29:07.5176050Z
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase does
> not satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5177452Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5179831Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5181277Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5182154Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5182951Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5183906Z
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase does not satisfy: only
> one of the following predicates match:
> 2023-09-17T00:29:07.5184769Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5185812Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5186880Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5187929Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5189073Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5190076Z
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase does not
> satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5190946Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5191983Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5192845Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5193532Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5194300Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated
[jira] [Closed] (FLINK-33219) Kafka connector has architecture test violation against Flink 1.18
[ https://issues.apache.org/jira/browse/FLINK-33219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-33219. -- Fix Version/s: kafka-3.1.0 Resolution: Fixed Fixed in: apache/flink-connector-kafka@main b09928d5ef290f2a046dc1fe40b4c5cebe76f997 > Kafka connector has architecture test violation against Flink 1.18 > -- > > Key: FLINK-33219 > URL: https://issues.apache.org/jira/browse/FLINK-33219 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: kafka-4.0.0 >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: kafka-3.1.0 > > > FLINK-31804 modified architecture rule about MiniCluster, which changes error > message and leads to Kafka connector's architecture test fail: > > [https://github.com/apache/flink-connector-kafka/actions/runs/6453974340/job/17518572377] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33219) Kafka connector has architecture test violation against Flink 1.18
[ https://issues.apache.org/jira/browse/FLINK-33219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-33219: -- Assignee: Qingsheng Ren > Kafka connector has architecture test violation against Flink 1.18 > -- > > Key: FLINK-33219 > URL: https://issues.apache.org/jira/browse/FLINK-33219 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: kafka-4.0.0 >Reporter: Qingsheng Ren >Assignee: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > FLINK-31804 modified architecture rule about MiniCluster, which changes error > message and leads to Kafka connector's architecture test fail: > > [https://github.com/apache/flink-connector-kafka/actions/runs/6453974340/job/17518572377] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-33217:
---
Attachment: UnnestNullErrorTest.scala
> Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
> -
>
> Key: FLINK-33217
> URL: https://issues.apache.org/jira/browse/FLINK-33217
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.3, 1.18.0, 1.19.0
>Reporter: Robert Metzger
>Priority: Major
> Attachments: UnnestNullErrorTest.scala
>
>
> Steps to reproduce:
> Take a column of type
> {code:java}
> business_data ARRAY
> {code}
> Take this query
> {code:java}
> select bd_name from reproduce_unnest LEFT JOIN
> UNNEST(reproduce_unnest.business_data) AS exploded_bd(bd_name) ON true
> {code}
> And get this error
> {code:java}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of rel before registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" bd_name) NOT NULL
> rowtype of rel after registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL f0) NOT NULL
> Difference:
> bd_name: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" -> VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:2206)
> at
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:275)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1270)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:598)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:498)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
> {code}
> I have implemented a small test case, which fails against Flink 1.15, 1.8 and
> the latest master branch.
> Workarounds:
> 1. Drop "NOT NULL" in array type
> 2. Drop "LEFT" from "LEFT JOIN".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773311#comment-17773311
]
Robert Metzger edited comment on FLINK-33217 at 10/9/23 1:31 PM:
-
Actually, you can make the reproducer even simpler:
{code}business_data ARRAY{code}
also fails with the same error. I updated the code & description again.
was (Author: rmetzger):
Actually, you can make the reproducer even simpler:
{code}business_data ARRAY{code}
also works. I updated the code & description again.
> Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
> -
>
> Key: FLINK-33217
> URL: https://issues.apache.org/jira/browse/FLINK-33217
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.3, 1.18.0, 1.19.0
>Reporter: Robert Metzger
>Priority: Major
> Attachments: UnnestNullErrorTest.scala
>
>
> Steps to reproduce:
> Take a column of type
> {code:java}
> business_data ARRAY
> {code}
> Take this query
> {code:java}
> select bd_name from reproduce_unnest LEFT JOIN
> UNNEST(reproduce_unnest.business_data) AS exploded_bd(bd_name) ON true
> {code}
> And get this error
> {code:java}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of rel before registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" bd_name) NOT NULL
> rowtype of rel after registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL f0) NOT NULL
> Difference:
> bd_name: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" -> VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:2206)
> at
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:275)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1270)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:598)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:498)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
> {code}
> I have implemented a small test case, which fails against Flink 1.15, 1.8 and
> the latest master branch.
> Workarounds:
> 1. Drop "NOT NULL" in array type
> 2. Drop "LEFT" from "LEFT JOIN".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-33217:
---
Attachment: (was: UnnestNullErrorTest-1.scala)
> Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
> -
>
> Key: FLINK-33217
> URL: https://issues.apache.org/jira/browse/FLINK-33217
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.3, 1.18.0, 1.19.0
>Reporter: Robert Metzger
>Priority: Major
> Attachments: UnnestNullErrorTest.scala
>
>
> Steps to reproduce:
> Take a column of type
> {code:java}
> business_data ARRAY
> {code}
> Take this query
> {code:java}
> select bd_name from reproduce_unnest LEFT JOIN
> UNNEST(reproduce_unnest.business_data) AS exploded_bd(bd_name) ON true
> {code}
> And get this error
> {code:java}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of rel before registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" bd_name) NOT NULL
> rowtype of rel after registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL f0) NOT NULL
> Difference:
> bd_name: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" -> VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:2206)
> at
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:275)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1270)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:598)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:498)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
> {code}
> I have implemented a small test case, which fails against Flink 1.15, 1.8 and
> the latest master branch.
> Workarounds:
> 1. Drop "NOT NULL" in array type
> 2. Drop "LEFT" from "LEFT JOIN".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-33141) Promentheus Sink Connector - Amazon Managed Prometheus Request Signer
[ https://issues.apache.org/jira/browse/FLINK-33141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh resolved FLINK-33141. - Resolution: Implemented Resolved by https://issues.apache.org/jira/browse/FLINK-33138 > Promentheus Sink Connector - Amazon Managed Prometheus Request Signer > - > > Key: FLINK-33141 > URL: https://issues.apache.org/jira/browse/FLINK-33141 > Project: Flink > Issue Type: Sub-task > Components: Connectors / AWS >Reporter: Lorenzo Nicora >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-33217:
---
Description:
Steps to reproduce:
Take a column of type
{code:java}
business_data ARRAY
{code}
Take this query
{code:java}
select bd_name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data) AS exploded_bd(bd_name) ON true
{code}
And get this error
{code:java}
Caused by: java.lang.AssertionError: Type mismatch:
rowtype of rel before registration: RecordType(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" bd_name) NOT NULL
rowtype of rel after registration: RecordType(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL f0) NOT NULL
Difference:
bd_name: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" -> VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL
at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:2206)
at
org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:275)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1270)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:598)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:498)
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
{code}
I have implemented a small test case, which fails against Flink 1.15, 1.8 and
the latest master branch.
Workarounds:
1. Drop "NOT NULL" in array type
2. Drop "LEFT" from "LEFT JOIN".
was:
Steps to reproduce:
Take a column of type
{code:java}
business_data ROW<`updateEvent` ARRAY>
{code}
Take this query
{code:java}
select ue_name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(ue_name) ON
true
{code}
And get this error
{code:java}
Caused by: java.lang.AssertionError: Type mismatch:
rowtype of rel before registration:
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" ue_name) NOT NULL
rowtype of rel after registration:
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL f0) NOT NULL
Difference:
ue_name: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" -> VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL
at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:2206)
at
org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:275)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1270)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:598)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
at
org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:498)
at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
at
org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
{code}
I have implemented a small test case, which fails against Flink 1.15, 1.8 and
the latest master branch.
> Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
> -
>
> Key: FLINK-33217
> URL: https://issues.apache.org/jira/browse/FLINK-33217
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.3, 1.18.0, 1.19.0
>Reporter: Robert Metzger
>Priority: Major
> Attachments: UnnestNullErrorTest-1.scala
>
>
> Steps to reproduce:
> Take a column of type
> {code:java}
> business_data ARRAY
> {code}
> Take this query
> {code:java}
> select bd_name from reproduce_unnest LEFT JOIN
> UNNEST(reproduce_unnest.business_data) AS exploded_bd(bd_name) ON true
> {code}
> And get this error
> {code:java}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of rel before registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" bd_name)
[jira] [Assigned] (FLINK-33138) Prometheus Connector Sink - DataStream API implementation
[ https://issues.apache.org/jira/browse/FLINK-33138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Liang Teoh reassigned FLINK-33138: --- Assignee: Lorenzo Nicora > Prometheus Connector Sink - DataStream API implementation > - > > Key: FLINK-33138 > URL: https://issues.apache.org/jira/browse/FLINK-33138 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Prometheus >Reporter: Lorenzo Nicora >Assignee: Lorenzo Nicora >Priority: Major > Fix For: prometheus-connector-1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33174) enabling tablesample bernoulli in flink
[ https://issues.apache.org/jira/browse/FLINK-33174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773306#comment-17773306 ] xiaogang zhou commented on FLINK-33174: --- [~libenchao] [~lsy] [~martijnvisser] Thanks all for your comments, Let me prepare a FLIP first and wait for the calcite upgrading > enabling tablesample bernoulli in flink > --- > > Key: FLINK-33174 > URL: https://issues.apache.org/jira/browse/FLINK-33174 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.17.1 >Reporter: xiaogang zhou >Priority: Major > > I'd like to introduce a table sample function to enable fast sampling to > streamings. > this is enlighted by https://issues.apache.org/jira/browse/CALCITE-5971 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33191) Kafka Connector should directly depend on 3rd-party libs instead of flink-shaded repo
[ https://issues.apache.org/jira/browse/FLINK-33191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33191: --- Labels: pull-request-available (was: ) > Kafka Connector should directly depend on 3rd-party libs instead of > flink-shaded repo > - > > Key: FLINK-33191 > URL: https://issues.apache.org/jira/browse/FLINK-33191 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Martijn Visser >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33191][Connector/Kafka] Remove dependency on Flink Shaded [flink-connector-kafka]
MartijnVisser opened a new pull request, #57: URL: https://github.com/apache/flink-connector-kafka/pull/57 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-33191) Kafka Connector should directly depend on 3rd-party libs instead of flink-shaded repo
[ https://issues.apache.org/jira/browse/FLINK-33191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-33191: -- Assignee: Martijn Visser > Kafka Connector should directly depend on 3rd-party libs instead of > flink-shaded repo > - > > Key: FLINK-33191 > URL: https://issues.apache.org/jira/browse/FLINK-33191 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Martijn Visser >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33046][FLIP-333] Redesign Apache Flink website [flink-web]
dannycranmer commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1753004143 @MartijnVisser thanks for reporting this: > Not sure if it's intended, but when hovering over the Flink logo in top left, then "Apache Flink" becomes visible (in dark mode). I think we should always display it, because the sudden appearance after a hover feels wrong? Fixed and deployed to the demo site. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-33217:
---
Attachment: UnnestNullErrorTest-1.scala
> Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
> -
>
> Key: FLINK-33217
> URL: https://issues.apache.org/jira/browse/FLINK-33217
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.3, 1.18.0, 1.19.0
>Reporter: Robert Metzger
>Priority: Major
> Attachments: UnnestNullErrorTest-1.scala, UnnestNullErrorTest.scala
>
>
> Steps to reproduce:
> Take a column of type
> {code:java}
> business_data ROW<`id` STRING, `updateEvent` ARRAY NULL> NOT NULL>> {code}
> Take this query
> {code:java}
> select id, ue_name from reproduce_unnest LEFT JOIN
> UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(ue_name) ON
> true {code}
> And get this error
> {code:java}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of rel before registration:
> RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" ue_name) NOT NULL
> rowtype of rel after registration:
> RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
> Difference:
> ue_name: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" -> VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:2206)
> at
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:275)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1270)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:598)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:498)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
> ... 66 more
> {code}
> I have implemented a small test case, which fails against Flink 1.15, 1.8 and
> the latest master branch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33175) Nightly builds from S3 are not available for download, breaking all connector tests
[
https://issues.apache.org/jira/browse/FLINK-33175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773294#comment-17773294
]
Hong Liang Teoh commented on FLINK-33175:
-
Verified the flink-connector-aws one succeeds as well. Thank you for looking
into this, everyone!
https://github.com/apache/flink-connector-aws/actions/runs/6450772599/job/17520032785
> Nightly builds from S3 are not available for download, breaking all connector
> tests
> ---
>
> Key: FLINK-33175
> URL: https://issues.apache.org/jira/browse/FLINK-33175
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Reporter: Martijn Visser
>Assignee: Jing Ge
>Priority: Blocker
>
> All downloads of Flink binaries fail with:
> {code:java}
> Run wget -q -c
> https://s3.amazonaws.com/flink-nightly/flink-1.18-SNAPSHOT-bin-scala_2.12.tgz
> -O - | tar -xz
> gzip: stdin: unexpected end of file
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> Error: Process completed with exit code 2.
> {code}
> This goes for 1.18, but also 1.17 and 1.16
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-32512] Don't register resource to user resource manager when creating temporary function [flink]
fsk119 commented on code in PR #22938:
URL: https://github.com/apache/flink/pull/22938#discussion_r1350236277
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/resource/ResourceManager.java:
##
@@ -65,9 +69,14 @@ public class ResourceManager implements Closeable {
private static final String FILE_SCHEME = "file";
private final Path localResourceDir;
+/** Resource infos for functions. */
+private final Map functionResourceInfos;
+
protected final Map resourceInfos;
protected final MutableURLClassLoader userClassLoader;
+private final List createdClassLoaderList;
Review Comment:
Why we need this list here? I find the only usage is here.
```
URLClassLoader classLoader =
resourceManager.createUserClassLoader(function.getFunctionResources());
try {
UserDefinedFunctionHelper.validateClass(
(Class)
classLoader.loadClass(function.getClassName()));
} finally {
resourceManager.closeUserClassLoader(classLoader);
}
```
If we close the created classloder immediately, can we move the management
outside?
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/resource/ResourceManager.java:
##
@@ -124,15 +136,79 @@ public String registerFileResource(ResourceUri
resourceUri) throws IOException {
Collections.singletonList(resourceUri),
ResourceType.FILE,
false,
-url -> {});
+url -> {},
+false);
registerResources(stagingResources, false);
return resourceInfos.get(new
ArrayList<>(stagingResources.keySet()).get(0)).getPath();
}
+/**
+ * Register a resource for function and add it to the function resource
infos. If the file is
+ * remote, it will be copied to a local file.
+ *
+ * @param resourceUris the resource uri for function.
+ */
+public void registerFunctionResources(Set resourceUris)
throws IOException {
+prepareStagingResources(
+resourceUris,
+ResourceType.JAR,
+true,
+url -> {
+try {
+JarUtils.checkJarFile(url);
+} catch (IOException e) {
+throw new ValidationException(
+String.format("Failed to register jar resource
[%s]", url), e);
+}
+},
+true);
+}
+
+/**
+ * Unregister the resource uri in function resources, when the reference
count of the resource
+ * is 0, the resource will be removed from the function resources.
+ *
+ * @param resourceUris the uris to unregister in function resources.
+ */
+public void unregisterFunctionResources(List resourceUris) {
+if (!resourceUris.isEmpty()) {
+resourceUris.forEach(
+uri -> {
+ResourceCounter counter =
functionResourceInfos.get(uri);
+if (counter != null && counter.decreaseCounter()) {
+functionResourceInfos.remove(uri);
+}
Review Comment:
No. We always assume table env is used by one thread at any moement.
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/resource/ResourceManager.java:
##
@@ -80,8 +89,10 @@ public ResourceManager(ReadableConfig config,
MutableURLClassLoader userClassLoa
new Path(
config.get(TableConfigOptions.RESOURCES_DOWNLOAD_DIR),
String.format("flink-table-%s", UUID.randomUUID()));
+this.functionResourceInfos = new HashMap<>();
this.resourceInfos = new HashMap<>();
this.userClassLoader = userClassLoader;
+this.createdClassLoaderList = new ArrayList<>();
Review Comment:
use a better name?
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/FunctionCatalog.java:
##
@@ -44,7 +44,11 @@
import org.apache.flink.table.resource.ResourceUri;
import org.apache.flink.util.Preconditions;
+import javax.annotation.Nullable;
Review Comment:
+1
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/FunctionCatalog.java:
##
@@ -569,6 +581,8 @@ private Optional
resolvePreciseFunctionReference(Object
CatalogFunction potentialResult =
tempCatalogFunctions.get(normalizedIdentifier);
if (potentialResult != null) {
+registerFunctionJarResources(
+oi.asSummaryString(),
potentialResult.getFunctionResources());
Review Comment:
for temporary function, Function
Re: [PR] [FLINK-33219][connector/kafka] Add new archunit violation messages introduced by FLINK-31804 [flink-connector-kafka]
boring-cyborg[bot] commented on PR #56: URL: https://github.com/apache/flink-connector-kafka/pull/56#issuecomment-1752946580 Awesome work, congrats on your first merged pull request! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33219][connector/kafka] Add new archunit violation messages introduced by FLINK-31804 [flink-connector-kafka]
MartijnVisser merged PR #56: URL: https://github.com/apache/flink-connector-kafka/pull/56 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-14102) Introduce DB2Dialect
[ https://issues.apache.org/jira/browse/FLINK-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser closed FLINK-14102. -- Fix Version/s: jdbc-3.2.0 Resolution: Fixed Merged in apache/flink-connector-jdbc:main @ 3ecd0fdacdc9d0a66b6de04182132eef821be401 > Introduce DB2Dialect > > > Key: FLINK-14102 > URL: https://issues.apache.org/jira/browse/FLINK-14102 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: Canbin Zheng >Assignee: Zhongqiang Gong >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: jdbc-3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-14102) Introduce DB2Dialect
[ https://issues.apache.org/jira/browse/FLINK-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martijn Visser reassigned FLINK-14102: -- Assignee: Zhongqiang Gong > Introduce DB2Dialect > > > Key: FLINK-14102 > URL: https://issues.apache.org/jira/browse/FLINK-14102 > Project: Flink > Issue Type: Sub-task > Components: Connectors / JDBC >Reporter: Canbin Zheng >Assignee: Zhongqiang Gong >Priority: Major > Labels: auto-unassigned, pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-14102] Introduce DB2Dialect. [flink-connector-jdbc]
MartijnVisser merged PR #8: URL: https://github.com/apache/flink-connector-jdbc/pull/8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773263#comment-17773263
]
Sergey Nuyanzin edited comment on FLINK-33217 at 10/9/23 11:52 AM:
---
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}},
also keep in mind type
{code:sql}
business_data ROW<`id` STRING, `updateEvent` ARRAY
NOT NULL>>
{code}
where it is called {{name}}
so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
2. However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent)
business_data, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT
NULL
set is rel#104:LogicalCorrelate.NONE.any.None:
0.[NONE].[NONE](left=HepRelVertex#98,right=HepRelVertex#103,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalCorrelate(correlation=[$cor0], joinType=[left],
requiredColumns=[{0}])
LogicalTableScan(table=[[default_catalog, default_database, reproduce_unnest,
source: [CollectionTableSource(business_data)]]])
LogicalTableFunctionScan(invocation=[$UNNEST_ROWS$1($cor0.business_data.updateEvent)],
rowType=[RecordType:peek_no_expand(VARCHAR(2147483647) name)])
{noformat}
Now if look at type comparison in trace
{noformat}
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
we can see that the difference is that for the first type column {{name}} is
nullable and for the second is not and this is probably should be fixed (not
sure whether it could be fixed with FLIP-154 or not, will try to double check
it later)
the WA is to ensure that there is the same type on both sides of the join e.g.
nullable for both
like
{code:sql}
CREATE TABLE reproduce_unnest (
business_data ROW<`id` STRING, `updateEvent` ARRAY NOT
NULL>>
) WITH (
'connector' = 'COLLECTION',
'is-bounded' = 'false'
)
{code}
and
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true
{code}
was (Author: sergey nuyanzin):
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}},
also keep in mind type
{code:sql}
business_data ROW<`id` STRING, `updateEvent` ARRAY
NOT NULL>>
{code}
where it is called {{name}}
so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is
[jira] [Comment Edited] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773263#comment-17773263
]
Sergey Nuyanzin edited comment on FLINK-33217 at 10/9/23 11:51 AM:
---
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}},
also keep in mind type
{code:sql}
business_data ROW<`id` STRING, `updateEvent` ARRAY
NOT NULL>>
{code}
where it is called {{name}}
so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent)
business_data, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT
NULL
set is rel#104:LogicalCorrelate.NONE.any.None:
0.[NONE].[NONE](left=HepRelVertex#98,right=HepRelVertex#103,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalCorrelate(correlation=[$cor0], joinType=[left],
requiredColumns=[{0}])
LogicalTableScan(table=[[default_catalog, default_database, reproduce_unnest,
source: [CollectionTableSource(business_data)]]])
LogicalTableFunctionScan(invocation=[$UNNEST_ROWS$1($cor0.business_data.updateEvent)],
rowType=[RecordType:peek_no_expand(VARCHAR(2147483647) name)])
{noformat}
Now if look at type comparison in trace
{noformat}
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
we can see that the difference is that for the first type column {{name}} is
nullable and for the second is not and this is probably should be fixed (not
sure whether it could be fixed with FLIP-154 or not, will try to double check
it later)
the WA is to ensure that there is the same type on both sides of the join e.g.
nullable for both
like
{code:sql}
CREATE TABLE reproduce_unnest (
business_data ROW<`id` STRING, `updateEvent` ARRAY NOT
NULL>>
) WITH (
'connector' = 'COLLECTION',
'is-bounded' = 'false'
)
{code}
and
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true
{code}
was (Author: sergey nuyanzin):
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}},
also keep in mind type
{code:sql}
business_data ROW<`id` STRING, `updateEvent` ARRAY
NOT NULL>>
{code}
where it is called {{name}}
so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is
[jira] [Comment Edited] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773263#comment-17773263
]
Sergey Nuyanzin edited comment on FLINK-33217 at 10/9/23 11:48 AM:
---
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}},
also keep in mind type
{code:sql}
business_data ROW<`id` STRING, `updateEvent` ARRAY
NOT NULL>>
{code}
where it is called {{name}}
so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent)
business_data, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT
NULL
set is rel#104:LogicalCorrelate.NONE.any.None:
0.[NONE].[NONE](left=HepRelVertex#98,right=HepRelVertex#103,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalCorrelate(correlation=[$cor0], joinType=[left],
requiredColumns=[{0}])
LogicalTableScan(table=[[default_catalog, default_database, reproduce_unnest,
source: [CollectionTableSource(business_data)]]])
LogicalTableFunctionScan(invocation=[$UNNEST_ROWS$1($cor0.business_data.updateEvent)],
rowType=[RecordType:peek_no_expand(VARCHAR(2147483647) name)])
{noformat}
Now if look at type comparison in trace
{noformat}
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
we can see that the difference is that for the first type column {{name}} is
nullable and for the second is not
the WA is to ensure that there is the same type on both sides of the join e.g.
nullable for both
like
{code:sql}
CREATE TABLE reproduce_unnest (
business_data ROW<`id` STRING, `updateEvent` ARRAY NOT
NULL>>
) WITH (
'connector' = 'COLLECTION',
'is-bounded' = 'false'
)
{code}
and
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true
{code}
was (Author: sergey nuyanzin):
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}}, so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression
Re: [PR] [FLINK-33046][FLIP-333] Redesign Apache Flink website [flink-web]
dannycranmer commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1752855653 > Open any Staging link in an incognito window. Using Version 117.0.5938.149 (Official Build) (arm64) Thanks, have reproduced in incognito! (weird) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773263#comment-17773263
]
Sergey Nuyanzin edited comment on FLINK-33217 at 10/9/23 11:45 AM:
---
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}}, so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent)
business_data, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT
NULL
set is rel#104:LogicalCorrelate.NONE.any.None:
0.[NONE].[NONE](left=HepRelVertex#98,right=HepRelVertex#103,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalCorrelate(correlation=[$cor0], joinType=[left],
requiredColumns=[{0}])
LogicalTableScan(table=[[default_catalog, default_database, reproduce_unnest,
source: [CollectionTableSource(business_data)]]])
LogicalTableFunctionScan(invocation=[$UNNEST_ROWS$1($cor0.business_data.updateEvent)],
rowType=[RecordType:peek_no_expand(VARCHAR(2147483647) name)])
{noformat}
Now if look at type comparison in trace
{noformat}
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
we can see that the difference is that for the first type column {{name}} is
nullable and for the second is not
the WA is to ensure that there is the same type on both sides of the join e.g.
nullable for both
like
{code:sql}
CREATE TABLE reproduce_unnest (
business_data ROW<`id` STRING, `updateEvent` ARRAY NOT
NULL>>
) WITH (
'connector' = 'COLLECTION',
'is-bounded' = 'false'
)
{code}
and
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true
{code}
was (Author: sergey nuyanzin):
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}}, so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression type is
Re: [PR] [FLINK-33046][FLIP-333] Redesign Apache Flink website [flink-web]
dannycranmer commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1752854785 > Hmmm. I guess I just miss the menu on the left. I don't like it but won't block it. We can always change it later if we get similar feedback. Thanks for not blocking -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773263#comment-17773263
]
Sergey Nuyanzin edited comment on FLINK-33217 at 10/9/23 11:44 AM:
---
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}}, so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent)
business_data, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT
NULL
set is rel#104:LogicalCorrelate.NONE.any.None:
0.[NONE].[NONE](left=HepRelVertex#98,right=HepRelVertex#103,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalCorrelate(correlation=[$cor0], joinType=[left],
requiredColumns=[{0}])
LogicalTableScan(table=[[default_catalog, default_database, reproduce_unnest,
source: [CollectionTableSource(business_data)]]])
LogicalTableFunctionScan(invocation=[$UNNEST_ROWS$1($cor0.business_data.updateEvent)],
rowType=[RecordType:peek_no_expand(VARCHAR(2147483647) name)])
{noformat}
Now if look at type comparison in trace
{noformat}
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
we can see that the difference is that for the first type column {{name}} is
nullable and for the second is not
the WA is to ensure that there is the same type on both sides of the join e.g.
nullable for both
like
{code:sql}
CREATE TABLE reproduce_unnest (
business_data ROW<`id` STRING, `updateEvent` ARRAY NOT
NULL>>
) WITH (
'connector' = 'COLLECTION',
'is-bounded' = 'false'
)
{code}
and
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true
{code}
was (Author: sergey nuyanzin):
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
rowtype of rel after registration:
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}}, so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
Re: [PR] [FLINK-33046][FLIP-333] Redesign Apache Flink website [flink-web]
dannycranmer commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1752852964 @MartijnVisser thanks for reporting this: > I think we just shouldn't use the distinguishment at all if you've already visited a link before. Fixed and deployed to the demo site. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33046][FLIP-333] Redesign Apache Flink website [flink-web]
dannycranmer commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1752851944 > I found a strange behavior that when I click the nav title on the right, it will hide the head of lines:  What I expect:  > > The problematic link https://website-refresh.d193kg429zpv7e.amplifyapp.com/what-is-flink/flink-applications/#state Thanks for reporting this! Fixed and deployed to the demo site. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773263#comment-17773263
]
Sergey Nuyanzin commented on FLINK-33217:
-
Hi [~rmetzger]
thanks for raising the issue
It looks like the problem consists of several parts
1. there is an error in SQL provided in description
it complains about type mismatch (in trace) like
{noformat}
RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL
name) NOT NULL ARRAY updateEvent) business_data, VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" ue_name) NOT NULL
rowtype of rel after registration:
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
the first discrepancy is naimng: in one column it is {{ue_name}} while in
another it is {{name}}, so the more correct SQL should look like
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true;
{code}
However this will also fail with
{noformat}
set type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER
SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
expression type is RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647)
CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent)
business_data, VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT
NULL
set is rel#104:LogicalCorrelate.NONE.any.None:
0.[NONE].[NONE](left=HepRelVertex#98,right=HepRelVertex#103,correlation=$cor0,joinType=left,requiredColumns={0})
expression is LogicalCorrelate(correlation=[$cor0], joinType=[left],
requiredColumns=[{0}])
LogicalTableScan(table=[[default_catalog, default_database, reproduce_unnest,
source: [CollectionTableSource(business_data)]]])
LogicalTableFunctionScan(invocation=[$UNNEST_ROWS$1($cor0.business_data.updateEvent)],
rowType=[RecordType:peek_no_expand(VARCHAR(2147483647) name)])
{noformat}
Now if look at type comparison in trace
{noformat}
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" name) NOT NULL
RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
"UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
VARCHAR(2147483647) CHARACTER SET "UTF-16LE" NOT NULL name) NOT NULL
{noformat}
we can see that the difference is that for the first type column {{name}} is
nullable and for the second is not
the WA is to ensure that there is the same type on both sides of the join e.g.
nullable for both
like
{code:sql}
CREATE TABLE reproduce_unnest (
business_data ROW<`id` STRING, `updateEvent` ARRAY NOT
NULL>>
) WITH (
'connector' = 'COLLECTION',
'is-bounded' = 'false'
)
{code}
and
{code:sql}
select id, name from reproduce_unnest LEFT JOIN
UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(name) ON true
{code}
> Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
> -
>
> Key: FLINK-33217
> URL: https://issues.apache.org/jira/browse/FLINK-33217
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.3, 1.18.0, 1.19.0
>Reporter: Robert Metzger
>Priority: Major
> Attachments: UnnestNullErrorTest.scala
>
>
> Steps to reproduce:
> Take a column of type
> {code:java}
> business_data ROW<`id` STRING, `updateEvent` ARRAY NULL> NOT NULL>> {code}
> Take this query
> {code:java}
> select id, ue_name from reproduce_unnest LEFT JOIN
> UNNEST(reproduce_unnest.business_data.updateEvent) AS exploded_ue(ue_name) ON
> true {code}
> And get this error
> {code:java}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of rel before registration:
> RecordType(RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" id, RecordType:peek_no_expand(VARCHAR(2147483647) CHARACTER SET
> "UTF-16LE" NOT NULL name) NOT NULL ARRAY updateEvent) business_data,
> VARCHAR(2147483647) CHARACTER SET "UTF-16LE" ue_name) NOT NULL
> rowtype of rel after registration:
> RecordType(RecordType:peek_no_expand(VARCHAR(2147483647)
[jira] [Created] (FLINK-33221) Add config options for administrator JVM options
Zhanghao Chen created FLINK-33221:
-
Summary: Add config options for administrator JVM options
Key: FLINK-33221
URL: https://issues.apache.org/jira/browse/FLINK-33221
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration
Reporter: Zhanghao Chen
We encounter similar issues described in SPARK-23472. Users may need to add JVM
options to their Flink applications (e.g. to tune GC options). They typically
use {{env.java.opts.x}} series of options to do so. We also have a set of
administrator JVM options to apply by default, e.g. to enable GC log, tune GC
options, etc. Both use cases will need to set the same series of options and
will clobber one another.
In the past, we generated and pretended to the administrator JVM options in the
Java code for generating the starting command for JM/TM. However, this has been
proven to be difficult to maintain.
Therefore, I propose to also add a set of default JVM options for administrator
use that prepends the user-set extra JVM options. We can mark the existing
{{env.java.opts.x}} series as user-set extra JVM options and add a set of new
{{env.java.opts.x.default}} options for administrator use.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33219][connector/kafka] Add new archunit violation messages introduced by FLINK-31804 [flink-connector-kafka]
MartijnVisser commented on code in PR #56: URL: https://github.com/apache/flink-connector-kafka/pull/56#discussion_r1350181266 ## .github/workflows/push_pr.yml: ## @@ -23,6 +23,10 @@ concurrency: cancel-in-progress: true jobs: compile_and_test: +strategy: + matrix: +flink: [ 1.17-SNAPSHOT, 1.18-SNAPSHOT ] Review Comment: ```suggestion flink: [ 1.17.1, 1.18-SNAPSHOT ] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33218] Replace glob with version parameter [flink]
flinkbot commented on PR #23498: URL: https://github.com/apache/flink/pull/23498#issuecomment-1752805131 ## CI report: * 809c9f8b56a6e2d8adef1fc905d3c2fe08731266 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33218) First Steps - error when run with zsh
[
https://issues.apache.org/jira/browse/FLINK-33218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33218:
---
Labels: pull-request-available (was: )
> First Steps - error when run with zsh
> -
>
> Key: FLINK-33218
> URL: https://issues.apache.org/jira/browse/FLINK-33218
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Reporter: Robin Moffatt
>Priority: Not a Priority
> Labels: pull-request-available
>
> If a user of zsh (the default on MacOS) runs the literal command that's given
> under "Browsing the project directory" they get an error:
> {code:java}
> $ cd flink-* && ls -l
> cd: string not in pwd: flink-1.17.1
> {code}
>
> This is because the behaviour of `cd` is different under zsh than bash and
> the glob triggers this. I've written up [an
> explanation|https://rmoff.net/2023/10/04/cd-string-not-in-pwd/] for those
> interested.
> IMO the fix is to hardcode the version in the instructions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-33218] Replace glob with version parameter [flink]
rmoff opened a new pull request, #23498: URL: https://github.com/apache/flink/pull/23498 ## What is the purpose of the change Fix a nit in the docs that will currently trip up zsh users See https://issues.apache.org/jira/browse/FLINK-33218 ## Brief change log * Use Hugo parameters to insert the versions in the generated text -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-25055) Support listen and notify mechanism for PartitionRequest
[ https://issues.apache.org/jira/browse/FLINK-25055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo reassigned FLINK-25055: -- Assignee: Yangze Guo (was: Fang Yong) > Support listen and notify mechanism for PartitionRequest > > > Key: FLINK-25055 > URL: https://issues.apache.org/jira/browse/FLINK-25055 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Affects Versions: 1.14.0, 1.12.5, 1.13.3 >Reporter: Fang Yong >Assignee: Yangze Guo >Priority: Major > Labels: pull-request-available, stale-assigned > > We submit batch jobs to flink session cluster with eager scheduler for olap. > JM deploys subtasks to TaskManager independently, and the downstream subtasks > may start before the upstream ones are ready. The downstream subtask sends > PartitionRequest to upstream ones, and may receive PartitionNotFoundException > from them. Then it will retry to send PartitionRequest after a few ms until > timeout. > The current approach raises two problems. First, there will be too many retry > PartitionRequest messages. Each downstream subtask will send PartitionRequest > to all its upstream subtasks and the total number of messages will be O(N*N), > where N is the parallelism of subtasks. Secondly, the interval between > polling retries will increase the delay for upstream and downstream tasks to > confirm PartitionRequest. > We want to support listen and notify mechanism for PartitionRequest when the > job needs no failover. Upstream TaskManager will add the PartitionRequest to > a listen list with a timeout checker, and notify the request when the task > register its partition in the TaskManager. > [~nkubicek] I noticed that your scenario of using flink is similar to ours. > What do you think? And hope to hear from you [~trohrmann] THX -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-25055][runtime] Support listen and notify mechanism for partition request [flink]
FangYongs closed pull request #19380: [FLINK-25055][runtime] Support listen and notify mechanism for partition request URL: https://github.com/apache/flink/pull/19380 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-32563) execute sanity checks only with Flink version that connectors were built against
[ https://issues.apache.org/jira/browse/FLINK-32563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot updated FLINK-32563: - Summary: execute sanity checks only with Flink version that connectors were built against (was: Allow connectors CI to specify the main supported Flink version) > execute sanity checks only with Flink version that connectors were built > against > > > Key: FLINK-32563 > URL: https://issues.apache.org/jira/browse/FLINK-32563 > Project: Flink > Issue Type: Technical Debt > Components: Build System / CI >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > > As part of [this > discussion|https://lists.apache.org/thread/pr0g812olzpgz21d9oodhc46db9jpxo3] > , the need for connectors to specify the main flink version that a connector > supports has arisen. > This CI variable will allow to configure the build and tests differently > depending on this version. This parameter would be optional. > The first use case is to run archunit tests only on the main supported > version as discussed in the above thread. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33171) Table SQL support Not Equal for TimePoint type and TimeString
[
https://issues.apache.org/jira/browse/FLINK-33171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773236#comment-17773236
]
Sergey Nuyanzin commented on FLINK-33171:
-
[~qingyue] thanks for ping
Currently I postponed a bit activity about FLIP-154 since I'm participating in
1.18.0 release activities.
I'm going to continue work on it once 1.18.0 is released
> Table SQL support Not Equal for TimePoint type and TimeString
> -
>
> Key: FLINK-33171
> URL: https://issues.apache.org/jira/browse/FLINK-33171
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.18.0, 1.17.1
>Reporter: Feng Jiajie
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.2, 1.18.1
>
>
> When executing the following SQL:
> {code:sql}
> SELECT
> time1,
> time1 = '2023-09-30 18:22:42.123' AS eq1,
> NOT (time1 = '2023-09-30 18:22:42.123') AS notEq1
> FROM table1;
> {code}
> the result is as follows:
> {code:java}
> ++-+++
> | op | time1 |eq1 | notEq1 |
> ++-+++
> | +I | 2023-09-30 18:22:42.123 | TRUE | TRUE |
> | +I | 2023-09-30 18:22:42.124 | FALSE | TRUE |
> ++-+++
> 2 rows in set
> {code}
> The "notEq1" in the first row should be FALSE.
> Here is the reproducing code:
> {code:java}
> import org.apache.flink.api.common.functions.RichMapFunction;
> import org.apache.flink.api.common.typeinfo.TypeInformation;
> import org.apache.flink.api.common.typeinfo.Types;
> import org.apache.flink.api.java.typeutils.RowTypeInfo;
> import org.apache.flink.configuration.Configuration;
> import org.apache.flink.streaming.api.datastream.DataStreamSource;
> import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
> import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
> import org.apache.flink.table.api.DataTypes;
> import org.apache.flink.table.api.Schema;
> import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
> import org.apache.flink.types.Row;
> public class TimePointNotEqualTest {
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment(new
> Configuration());
> env.setParallelism(1);
> DataStreamSource longDataStreamSource = env.fromSequence(0, 1);
> RowTypeInfo rowTypeInfo =
> new RowTypeInfo(new TypeInformation[] {Types.LONG}, new
> String[] {"time1"});
> SingleOutputStreamOperator map =
> longDataStreamSource.map(new RichMapFunction() {
> @Override
> public Row map(Long value) {
> Row row = new Row(1);
> row.setField(0, 1696069362123L + value);
> return row;
> }
> }, rowTypeInfo);
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
> Schema schema = Schema.newBuilder()
> .column("time1",
> DataTypes.TIMESTAMP_LTZ(3).bridgedTo(Long.class))
> .build();
> tableEnv.createTemporaryView("table1", map, schema);
> tableEnv.sqlQuery("SELECT "
> + "time1," // 2023-09-30 18:22:42.123
> + "time1 = '2023-09-30 18:22:42.123' AS eq1," // expect TRUE
> + "NOT (time1 = '2023-09-30 18:22:42.123') AS notEq1 " //
> expect FALSE but TRUE
> + "FROM table1").execute().print();
> }
> }
> {code}
> I would like to attempt to fix this issue. If possible, please assign the
> issue to me. Thank you.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33182) Allow metadata columns in NduAnalyzer with ChangelogNormalize
[ https://issues.apache.org/jira/browse/FLINK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773229#comment-17773229 ] Timo Walther edited comment on FLINK-33182 at 10/9/23 10:22 AM: [~lincoln.86xy] not sure if I understand your comment correctly. I understand that -D is dangerous and could cause non-determinism. However, ChangelogNormalize does not use the message value of -D. It reconstructs the -D from the previously stored row as can be seen in `DeduplicateFunctionHelper#processLastRowOnChangelog`. So metadata columns cannot affect determinism in this case. Note that only the primary key is used for equality. Metadata columns are not supported as primary key columns. was (Author: twalthr): [~lincoln.86xy] not sure if I understand your comment correctly. I understand that -D is dangerous and could cause non-determinism. However, ChangelogNormalize does not use the message value of -D. It reconstructs the -D from the previously stored row as can be seen in `DeduplicateFunctionHelper#processLastRowOnChangelog`. So metadata columns cannot affect determinism in this case. > Allow metadata columns in NduAnalyzer with ChangelogNormalize > - > > Key: FLINK-33182 > URL: https://issues.apache.org/jira/browse/FLINK-33182 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > Currently, the NduAnalyzer is very strict about metadata columns in updating > sources. However, for upsert sources (like Kafka) that contain an incomplete > changelog, the planner always adds a ChangelogNormalize node. > ChangelogNormalize will make sure that metadata columns can be considered > deterministic. So the NduAnalyzer should be satisfied in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33182) Allow metadata columns in NduAnalyzer with ChangelogNormalize
[ https://issues.apache.org/jira/browse/FLINK-33182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17773229#comment-17773229 ] Timo Walther commented on FLINK-33182: -- [~lincoln.86xy] not sure if I understand your comment correctly. I understand that -D is dangerous and could cause non-determinism. However, ChangelogNormalize does not use the message value of -D. It reconstructs the -D from the previously stored row as can be seen in `DeduplicateFunctionHelper#processLastRowOnChangelog`. So metadata columns cannot affect determinism in this case. > Allow metadata columns in NduAnalyzer with ChangelogNormalize > - > > Key: FLINK-33182 > URL: https://issues.apache.org/jira/browse/FLINK-33182 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Priority: Major > > Currently, the NduAnalyzer is very strict about metadata columns in updating > sources. However, for upsert sources (like Kafka) that contain an incomplete > changelog, the planner always adds a ChangelogNormalize node. > ChangelogNormalize will make sure that metadata columns can be considered > deterministic. So the NduAnalyzer should be satisfied in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33097][autoscaler] Initialize the generic autoscaler module and interfaces [flink-kubernetes-operator]
mxm commented on code in PR #677:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/677#discussion_r1350028811
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/config/AutoScalerOptions.java:
##
@@ -166,6 +197,13 @@ private static ConfigOptions.OptionBuilder
autoScalerConfig(String key) {
.stringType()
.asList()
.defaultValues()
+
.withDeprecatedKeys(deprecatedOperatorConfigKey("vertex.exclude.ids"))
.withDescription(
"A (semicolon-separated) list of vertex ids in
hexstring for which to disable scaling. Caution: For non-sink vertices this
will still scale their downstream operators until
https://issues.apache.org/jira/browse/FLINK-31215 is implemented.");
+
+public static final ConfigOption FLINK_CLIENT_TIMEOUT =
+autoScalerConfig("flink.client.timeout")
+.durationType()
+.defaultValue(Duration.ofSeconds(10))
+.withDescription("The timeout for waiting the flink rest
client to return.");
Review Comment:
Do we need to expose this as a configuration or can we just use the default?
I think it is better to let the user configure standard Flink configs as
listed here:
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#client-timeout
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/config/AutoScalerOptions.java:
##
@@ -15,149 +15,180 @@
* limitations under the License.
*/
-package org.apache.flink.kubernetes.operator.autoscaler.config;
+package org.apache.flink.autoscaler.config;
+import org.apache.flink.autoscaler.metrics.MetricAggregator;
import org.apache.flink.configuration.ConfigOption;
import org.apache.flink.configuration.ConfigOptions;
-import
org.apache.flink.kubernetes.operator.autoscaler.metrics.MetricAggregator;
import java.time.Duration;
import java.util.List;
-import static
org.apache.flink.kubernetes.operator.config.KubernetesOperatorConfigOptions.operatorConfig;
-
/** Config options related to the autoscaler module. */
public class AutoScalerOptions {
+public static final String DEPRECATED_K8S_OP_CONF_PREFIX =
"kubernetes.operator.";
+public static final String AUTOSCALER_CONF_PREFIX = "job.autoscaler.";
+
+private static String deprecatedOperatorConfigKey(String key) {
+return DEPRECATED_K8S_OP_CONF_PREFIX + AUTOSCALER_CONF_PREFIX + key;
+}
+
+private static String autoScalerConfigKey(String key) {
+return AUTOSCALER_CONF_PREFIX + key;
+}
+
private static ConfigOptions.OptionBuilder autoScalerConfig(String key) {
-return operatorConfig("job.autoscaler." + key);
+return ConfigOptions.key(autoScalerConfigKey(key));
}
public static final ConfigOption AUTOSCALER_ENABLED =
autoScalerConfig("enabled")
.booleanType()
.defaultValue(false)
+.withDeprecatedKeys(deprecatedOperatorConfigKey("enabled"))
Review Comment:
This might confuse some existing users because the deprecated keys will not
appear on the configuration page. Can we add a note on the configuration page
that we renamed the configuration prefix?
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/JobVertexScaler.java:
##
@@ -38,42 +36,42 @@
import java.util.Map;
import java.util.SortedMap;
-import static
org.apache.flink.kubernetes.operator.autoscaler.config.AutoScalerOptions.MAX_SCALE_DOWN_FACTOR;
-import static
org.apache.flink.kubernetes.operator.autoscaler.config.AutoScalerOptions.MAX_SCALE_UP_FACTOR;
-import static
org.apache.flink.kubernetes.operator.autoscaler.config.AutoScalerOptions.SCALE_UP_GRACE_PERIOD;
-import static
org.apache.flink.kubernetes.operator.autoscaler.config.AutoScalerOptions.TARGET_UTILIZATION;
-import static
org.apache.flink.kubernetes.operator.autoscaler.config.AutoScalerOptions.VERTEX_MAX_PARALLELISM;
-import static
org.apache.flink.kubernetes.operator.autoscaler.config.AutoScalerOptions.VERTEX_MIN_PARALLELISM;
-import static
org.apache.flink.kubernetes.operator.autoscaler.metrics.ScalingMetric.EXPECTED_PROCESSING_RATE;
-import static
org.apache.flink.kubernetes.operator.autoscaler.metrics.ScalingMetric.MAX_PARALLELISM;
-import static
org.apache.flink.kubernetes.operator.autoscaler.metrics.ScalingMetric.PARALLELISM;
-import static
org.apache.flink.kubernetes.operator.autoscaler.metrics.ScalingMetric.TRUE_PROCESSING_RATE;
+import static
org.apache.flink.autoscaler.config.AutoScalerOptions.MAX_SCALE_DOWN_FACTOR;
+import static
org.apache.flink.autoscaler.config.AutoScalerOptions.MAX_SCALE_UP_FACTOR;
+import static
org.apache.flink.autoscaler.config.AutoScalerOptions.SCALE_UP_GRACE_PERIOD;
+import static
Re: [PR] [FLINK-33097][autoscaler] Initialize the generic autoscaler module and interfaces [flink-kubernetes-operator]
mxm commented on PR #677: URL: https://github.com/apache/flink-kubernetes-operator/pull/677#issuecomment-1752727711 > If it's still needed, I can restore the `flink-kubernetes-operator-autoscaler` module. And moving all kubernetes-autoscaler related classes to this module. WDYT? Given that the autoscaler module is now decoupled from the operator version, it would make more sense than ever to have it pluggable. But given that the backends are not pluggable, the pluggability wouldn't be as useful anymore. I'm ok with removing the support for now if it proves difficult to maintain it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33219][connector/kafka] Add new archunit violation messages introduced by FLINK-31804 [flink-connector-kafka]
PatrickRen opened a new pull request, #56: URL: https://github.com/apache/flink-connector-kafka/pull/56 The reason we add new violation messages instead of update existing ones is that the patch of FLINK-31804 is only applied after Flink 1.18. We need to make sure the CI could run successfully for Flink versions before and after that. If Kafka connector decides to drop support for versions before 1.18 in the future, please re-freeze the violations then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33219) Kafka connector has architecture test violation against Flink 1.18
[ https://issues.apache.org/jira/browse/FLINK-33219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33219: --- Labels: pull-request-available (was: ) > Kafka connector has architecture test violation against Flink 1.18 > -- > > Key: FLINK-33219 > URL: https://issues.apache.org/jira/browse/FLINK-33219 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka >Affects Versions: kafka-4.0.0 >Reporter: Qingsheng Ren >Priority: Major > Labels: pull-request-available > > FLINK-31804 modified architecture rule about MiniCluster, which changes error > message and leads to Kafka connector's architecture test fail: > > [https://github.com/apache/flink-connector-kafka/actions/runs/6453974340/job/17518572377] -- This message was sent by Atlassian Jira (v8.20.10#820010)
