Re: [PR] [FLINK-34034][docs] Update the query hint docs to clarify the resolution of conflicts in kv hint and list hint [flink]
xuyangzhong commented on code in PR #24077:
URL: https://github.com/apache/flink/pull/24077#discussion_r1467317080
##
docs/content.zh/docs/dev/table/sql/queries/hints.md:
##
@@ -108,6 +108,39 @@ hintOption:
| stringLiteral
```
+### 查询提示使用中的冲突
+ Key-value 类型查询提示的冲突处理
+Key-value 类型的查询提示使用如下语法：
+
+```sql
+hintName '(' optionKey '=' optionVal [, optionKey '=' optionVal ]* ')'
+```
+
+当 Key-value 类型的查询提示发生冲突时，Flink 会按顺序用后定义的查询提示覆盖前面定义的同名查询提示。
+对于下面的例子，包含同相同 key 的查询提示 'max-attempts'：
Review Comment:
What about `在一个Key-value 类型的查询提示中存在重复的key时，Flink 会按顺序用后面的value覆盖前面的value`？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34113) Update flink-connector-elasticsearch to be compatible with updated SinkV2 interfaces
[ https://issues.apache.org/jira/browse/FLINK-34113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811151#comment-17811151 ] Jiabao Sun commented on FLINK-34113: [~martijnvisser] We can use TestSinkInitContext instead of MockInitContext to resolve this compilation problem. It is possible to make the elasticsearch connector still compatible with version 1.18. > Update flink-connector-elasticsearch to be compatible with updated SinkV2 > interfaces > > > Key: FLINK-34113 > URL: https://issues.apache.org/jira/browse/FLINK-34113 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / ElasticSearch >Reporter: Martijn Visser >Priority: Blocker > Labels: pull-request-available > Fix For: elasticsearch-3.2.0 > > > Make sure that the connector is updated to deal with the new changes > introduced in FLINK-33973 > See also > https://github.com/apache/flink-connector-elasticsearch/actions/runs/7539688654/job/20522689108#step:14:159 > for details on the current failure > This means that the new Elasticsearch connector will be compatible only with > Flink 1.19, with (the upcoming) v3.1.0 being compatible with only Flink 1.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34113) Update flink-connector-elasticsearch to be compatible with updated SinkV2 interfaces
[ https://issues.apache.org/jira/browse/FLINK-34113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34113: --- Labels: pull-request-available (was: ) > Update flink-connector-elasticsearch to be compatible with updated SinkV2 > interfaces > > > Key: FLINK-34113 > URL: https://issues.apache.org/jira/browse/FLINK-34113 > Project: Flink > Issue Type: Technical Debt > Components: Connectors / ElasticSearch >Reporter: Martijn Visser >Priority: Blocker > Labels: pull-request-available > Fix For: elasticsearch-3.2.0 > > > Make sure that the connector is updated to deal with the new changes > introduced in FLINK-33973 > See also > https://github.com/apache/flink-connector-elasticsearch/actions/runs/7539688654/job/20522689108#step:14:159 > for details on the current failure > This means that the new Elasticsearch connector will be compatible only with > Flink 1.19, with (the upcoming) v3.1.0 being compatible with only Flink 1.18 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34113] Update flink-connector-elasticsearch to be compatible with updated SinkV2 interfaces [flink-connector-elasticsearch]
boring-cyborg[bot] commented on PR #88: URL: https://github.com/apache/flink-connector-elasticsearch/pull/88#issuecomment-1911606536 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34034][docs] Update the query hint docs to clarify the resolution of conflicts in kv hint and list hint [flink]
xuyangzhong commented on code in PR #24077:
URL: https://github.com/apache/flink/pull/24077#discussion_r1467302542
##
docs/content/docs/dev/table/sql/queries/hints.md:
##
@@ -114,6 +114,42 @@ hintOption:
| stringLiteral
```
+### Conflict Cases In Query Hints
+ Resolution of Key-value Hint Conflicts
+For key-value hints, which are provided in the following syntax:
+
+```sql
+hintName '(' optionKey '=' optionVal [, optionKey '=' optionVal ]* ')'
+```
+
+When Flink encounters conflicting in key-value hints, it adopts a
last-write-wins strategy. This means that
+if multiple hint values are provided for the same key, Flink will use the
value from the last hint specified
+in the query. For instance, consider the following SQL query with conflicting
'max-attempts' values in the LOOKUP hint:
+
+```sql
+SELECT /*+ LOOKUP('table'='D', 'max-attempts'='3', 'max-attempts'='4') */ *
FROM t1 T JOIN t2 AS OF T.proctime AS D ON T.id = D.id;
+```
+
+In this case, Flink will resolve the conflict by selecting the last specified
value for 'max-attempts'.
+Therefore, the effective hint for 'max-attempts' will be '4'.
+
+ Resolution of List Hint Conflicts
+List hints are provided using the following syntax:
+
+```sql
+hintName '(' hintOption [, hintOption ]* ')'
+```
+
+With list hints, Flink resolves conflicts by adopting a first-accept strategy.
This means that the
+first specified hint in the list will take precedence and be effective.
+For example, consider the following SQL query with conflicting BROADCAST hints:
+```sql
+SELECT /*+ BROADCAST(t2, t1), BROADCAST(t1, t2) */ * FROM t1;
Review Comment:
ditto.
##
docs/content.zh/docs/dev/table/sql/queries/hints.md:
##
@@ -108,6 +108,39 @@ hintOption:
| stringLiteral
```
+### 查询提示使用中的冲突
+ Key-value 类型查询提示的冲突处理
+Key-value 类型的查询提示使用如下语法：
+
+```sql
+hintName '(' optionKey '=' optionVal [, optionKey '=' optionVal ]* ')'
+```
+
+当 Key-value 类型的查询提示发生冲突时，Flink 会按顺序用后定义的查询提示覆盖前面定义的同名查询提示。
+对于下面的例子，包含同相同 key 的查询提示 'max-attempts'：
+
+```sql
+SELECT /*+ LOOKUP('table'='D', 'max-attempts'='3', 'max-attempts'='4') */ *
FROM t1 T JOIN t2 AS OF T.proctime AS D ON T.id = D.id;
+```
+
+在这个例子里，Flink 会选择 'max-attempts' = '4' 的查询提示覆盖 'max-attempts' = '3' 的提示，
+所以最后 'max-attempts' 的值为 4。
+
+ List 类型查询提示的冲突处理
+List 类型的查询提示使用如下语法：
+
+```sql
+hintName '(' hintOption [, hintOption ]* ')'
+```
+
+对于 List 类型的查询提示，Flink 会选择最先被采纳的查询提示。如下面具有相同 BROADCAST 提示的例子：
+
+```sql
+SELECT /*+ BROADCAST(t2, t1), BROADCAST(t1, t2) */ * FROM t1;
Review Comment:
The hint is invalid actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-32743][Connectors/Kafka] Parse data from kafka connect and convert it into regular JSON data [flink-connector-kafka]
sunxiaojian commented on PR #42: URL: https://github.com/apache/flink-connector-kafka/pull/42#issuecomment-1911592095 > > Thanks for your review, it has been processed > > CI is failing, will you take a look? @MartijnVisser fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][docs] Fix scan.bounded.mode missing forward options [flink-connector-kafka]
Tan-JiaLiang commented on PR #60: URL: https://github.com/apache/flink-connector-kafka/pull/60#issuecomment-1911588128 @MartijnVisser Sorry late. > Do we have tests for them? To be honest, I checked all connectors(e.g. jdbc/hbase/filesystem...) but there is no test case for forward options. Forward options is a set of ConfigOption that are directly forwarded to the runtime implementation but don't affect the final execution topology. And according to the design of FLINK-24456, `scan.bounded.mode`, `scan.bounded.specific-offsets`, `scan.bounded.timestamp-millis` should be forward options. > And can you also check the Chinese docs? Yes, I filled in the gaps in the Chinese documentation and translated the content. Finally, the `sink.parallelism` option should not be a forward option, as it can lead to inconsistent parallelism between upstream and downstream operators and change the execution topology. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34209] Migrate FileSink to the new SinkV2 API [flink]
Jiabao-Sun commented on code in PR #24180:
URL: https://github.com/apache/flink/pull/24180#discussion_r1467184292
##
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/sink/writer/TestSinkInitContext.java:
##
@@ -48,7 +48,7 @@
import java.util.concurrent.ScheduledFuture;
/** A mock implementation of a {@code Sink.InitContext} to be used in sink
unit tests. */
-public class TestSinkInitContext implements Sink.InitContext {
+public class TestSinkInitContext implements WriterInitContext {
Review Comment:
~Hi @pvary, can we revert this change and provide a new class implements
`WriterInitContext`?
It may cause compatibility issues of the test.~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34053][table-planner] Support state ttl hint for group aggregate [flink]
xuyangzhong commented on code in PR #24179:
URL: https://github.com/apache/flink/pull/24179#discussion_r1467286142
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/RelTreeWriterImpl.scala:
##
@@ -149,7 +149,16 @@ class RelTreeWriterImpl(
if (stateTtlHints.nonEmpty) {
printValues.add(Pair.of("stateTtlHints",
RelExplainUtil.hintsToString(stateTtlHints)))
}
-
+case agg: Aggregate =>
+ val aggHints = FlinkHints.getAllStateTtlHints(agg.getHints)
+ if (aggHints.nonEmpty) {
+printValues.add(Pair.of("stateTtlHints",
RelExplainUtil.hintsToString(aggHints)))
+ }
+case agg: StreamPhysicalGroupAggregateBase =>
+ val aggHints = FlinkHints.getAllStateTtlHints(agg.hints)
+ if (aggHints.nonEmpty) {
+printValues.add(Pair.of("stateTtlHints",
RelExplainUtil.hintsToString(aggHints)))
+ }
Review Comment:
Good idea! I'll update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34222][table] End-to-end implementation of minibatch join [flink]
xishuaidelin commented on PR #24161: URL: https://github.com/apache/flink/pull/24161#issuecomment-1911579714 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33263][bugfix][table-planner] Update the flink shaded guava version used to 32 [flink]
flinkbot commented on PR #24200: URL: https://github.com/apache/flink/pull/24200#issuecomment-1911578061 ## CI report: * 5b43ccb1014c5a8be9ae9a19470b38877beeadb2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34216) Support fine-grained configuration to control filter push down for MongoDB Connector
[ https://issues.apache.org/jira/browse/FLINK-34216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34216: --- Labels: pull-request-available (was: ) > Support fine-grained configuration to control filter push down for MongoDB > Connector > > > Key: FLINK-34216 > URL: https://issues.apache.org/jira/browse/FLINK-34216 > Project: Flink > Issue Type: Sub-task > Components: Connectors / MongoDB >Affects Versions: mongodb-1.0.2 >Reporter: Jiabao Sun >Priority: Major > Labels: pull-request-available > Fix For: mongodb-1.1.0 > > > Support fine-grained configuration to control filter push down for MongoDB > Connector. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34216][connectors/mongodb] FLIP-377: Support fine-grained configuration to control filter push down for MongoDB Connector [flink-connector-mongodb]
Jiabao-Sun opened a new pull request, #23: URL: https://github.com/apache/flink-connector-mongodb/pull/23 [FLINK-34216][connectors/mongodb] FLIP-377: Support fine-grained configuration to control filter push down for MongoDB Connector This improvement implements [FLIP-377 Support fine-grained configuration to control filter push down for Table/SQL Sources](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768) This FLIP has 2 goals: - Introduces a new configuration filter.handling.policy to the JDBC and MongoDB connector. - Suggests a convention option name if other connectors are going to add an option for the same purpose. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34059][docs][table] Add documentation on STATE_TTL hint [flink]
xuyangzhong commented on code in PR #24198:
URL: https://github.com/apache/flink/pull/24198#discussion_r1467276024
##
docs/content/docs/dev/table/sql/queries/hints.md:
##
@@ -597,6 +597,131 @@ SELECT /*+ BROADCAST(t1) SHUFFLE_HASH(t1) */ * FROM t1
FULL OUTER JOIN t2 ON t1.
SELECT /*+ BROADCAST(t1) SHUFFLE_HASH(t1) */ * FROM t1 FULL OUTER JOIN t2 ON
t1.id > t2.id;
```
+### State TTL Hints
+
+{{< label Streaming >}}
+
+For stateful computation [Regular Join]({{< ref
"docs/dev/table/sql/queries/joins" >}}#regular-joins)
+and [Group Aggregation]({{< ref "docs/dev/table/sql/queries/group-agg" >}}),
users can
+use `STATE_TTL` hint to
+specify operator-level [Idle State Retention Time]({{< ref
"docs/dev/table/concepts/overview" >}}#idle-state-retention-time),
+which enables the aforementioned operators to have a different TTL against the
pipeline level configuration [table.exec.state.ttl]({{< ref
"docs/dev/table/config" >}}#table-exec-state-ttl).
+
+# Regular Join Examples
+
+```sql
+CREATE TABLE orders (
+ o_orderkey INT,
+ o_custkey INT,
+ o_status BOOLEAN,
+ o_totalprice DOUBLE
+) WITH (...);
+
+CREATE TABLE lineitem (
+ l_linenumber int,
+ l_orderkey int,
+ l_partkey int,
+ l_extendedprice double
+) WITH (...);
+
+CREATE TABLE customers (
+ c_custkey int,
+ c_address string
+) WITH (...);
+
+-- table name as hint key
+SELECT /*+ STATE_TTL('orders'='3d', 'lineitem'='1d') */ * FROM
+orders LEFT JOIN lineitem
+ON orders.o_orderkey = lineitem.l_orderkey;
+
+
+-- table alias as hint key
+SELECT /*+ STATE_TTL('o'='3d', 'l'='1d') */ * FROM
+orders o LEFT JOIN lineitem l
+ON o.o_orderkey = l.l_orderkey;
+
+-- temporary view name as hint key
+CREATE TEMPORARY VIEW left_input AS SELECT ... FROM orders WHERE ...;
+CREATE TEMPORARY VIEW right_input AS SELECT ... FROM lineitem WHERE ...;
+SELECT /*+ STATE_TTL('left_input'= '36s', 'right_input' = '15h') */ *
+FROM left_input JOIN right_input
+ON left_input.join_key = right_input.join_key;
+
+-- cascade joins
+SELECT /*+ STATE_TTL('o' = '3d', 'l' = '1d', 'c' = '10d') */ *
+FROM orders o LEFT OUTER JOIN lineitem l
+ON o.o_orderkey = l.l_orderkey
+LEFT OUTER JOIN customers c
+ON o.o_custkey = c.c_custkey;
+```
+
+# Group Aggregation Examples
+
+```sql
+-- table name as hint key
+SELECT /*+ STATE_TTL('orders' = '1d') */ o_orderkey, SUM(o_totalprice) AS
revenue
+FROM orders
+GROUP BY o_orderkey;
+
+-- table alias as hint key
+SELECT /*+ STATE_TTL('o' = '1d') */ o_orderkey, SUM(o_totalprice) AS revenue
+FROM orders AS o
+GROUP BY o_orderkey;
+
+-- query block alias as hint key
+SELECT /*+ STATE_TTL('tmp' = '1d') */ o_orderkey, SUM(o_totalprice) AS revenue
+FROM (SELECT o_orderkey, o_totalprice
+ FROM orders
+ WHERE o_shippriority = 0) tmp
+GROUP BY o_orderkey;
+```
+
+{{< hint info >}}
+Note:
+
+- Users can choose either table/view name or table alias as the hint key.
However, once the alias is specified, the `STATE_TTL` must be hinted on the
alias.
+- For cascade joins, the specified state TTLs will be interpreted as the left
and right state TTL for the first join operator and
+ the right state TTL for the second join operator (from a bottom-up order).
+ The left state TTL for the second join operator will be retrieved from the
configuration `table.exec.state.ttl`.
+ If users need to set a specific TTL value for the left state of the second
join operator, the query needs to be split into query blocks like
+ ```sql
+ CREATE TEMPORARY VIEW V AS
+ SELECT /*+ STATE_TTL('A' = '${ttl_A}', 'B' = '${ttl_B}')*/ * FROM A JOIN B
ON...;
+ SELECT /*+ STATE_TTL('V' = '${ttl_V}', 'C' = '${ttl_C}')*/ * FROM V JOIN C
ON ...;
+ ```
+- STATE_TTL hint only applies on the underlying query block.
+ {{< /hint >}}
+
+ Different ways to configure state TTL for SQL pipeline
Review Comment:
I feel placing this content here is somewhat awkward. How about moving it to
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/overview/#configure-operator-level-state-ttl
, and then adding a link at the beginning where `State TTL Hints` is
introduced?
##
docs/content.zh/docs/dev/table/sql/queries/hints.md:
##
@@ -549,6 +549,122 @@ SELECT /*+ BROADCAST(t1) SHUFFLE_HASH(t1) */ * FROM t1
FULL OUTER JOIN t2 ON t1.
-- 由于指定的两种联接提示都不支持不等值的联接条件。所以，只能使用支持非等值联接条件的 nested loop join。
SELECT /*+ BROADCAST(t1) SHUFFLE_HASH(t1) */ * FROM t1 FULL OUTER JOIN t2 ON
t1.id > t2.id;
```
+### 状态生命周期提示
+
+{{< label Streaming >}}
+
+对于有状态计算的[流连接]({{< ref "docs/dev/table/sql/queries/joins"
>}}#regular-joins)和[分组聚合]({{< ref "docs/dev/table/sql/queries/group-agg"
>}})操作，用户可以通过 `STATE_TTL` 来指定算子粒度的[空闲状态维持时间]({{< ref
"docs/dev/table/concepts/overview"
>}}#idle-state-retention-time)，该方式能够使得在上述状态算子中使用与 [table.exec.state.ttl]({{<
ref "docs/dev/table/config" >}}#table-exec-state-ttl) 不同的值。
+
+# 流连接示例
+
+```sql
+CREATE TABLE orders (
+ o_orderkey INT,
+ o_custkey INT,
+ o_status BOOLEAN,
+
Re: [PR] [FLINK-34053][table-planner] Support state ttl hint for group aggregate [flink]
LadyForest commented on code in PR #24179:
URL: https://github.com/apache/flink/pull/24179#discussion_r1467207831
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/utils/RelTreeWriterImpl.scala:
##
@@ -149,7 +149,16 @@ class RelTreeWriterImpl(
if (stateTtlHints.nonEmpty) {
printValues.add(Pair.of("stateTtlHints",
RelExplainUtil.hintsToString(stateTtlHints)))
}
-
+case agg: Aggregate =>
+ val aggHints = FlinkHints.getAllStateTtlHints(agg.getHints)
+ if (aggHints.nonEmpty) {
+printValues.add(Pair.of("stateTtlHints",
RelExplainUtil.hintsToString(aggHints)))
+ }
+case agg: StreamPhysicalGroupAggregateBase =>
+ val aggHints = FlinkHints.getAllStateTtlHints(agg.hints)
+ if (aggHints.nonEmpty) {
+printValues.add(Pair.of("stateTtlHints",
RelExplainUtil.hintsToString(aggHints)))
+ }
Review Comment:
Nit: what about
```scala
case _: Aggregate | _: StreamPhysicalGroupAggregateBase =>
val aggHints =
rel match {
case aggregate: Aggregate => aggregate.getHints
case _ =>
rel.asInstanceOf[StreamPhysicalGroupAggregateBase].hints
}
if (aggHints.nonEmpty) {
printValues.add(Pair.of("stateTtlHints",
RelExplainUtil.hintsToString(aggHints)))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33263][bugfix][table-planner] Update the flink shaded guava version used to 32 [flink]
BIOINSu commented on PR #24200: URL: https://github.com/apache/flink/pull/24200#issuecomment-1911564412 Hi @libenchao , this PR is a hotfix for the wrong version usage of flink shaded guava and change the version to 32. Please review this commit. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34214) FLIP-377: Support fine-grained configuration to control filter push down for Table/SQL Sources
[ https://issues.apache.org/jira/browse/FLINK-34214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811142#comment-17811142 ] Rascal Wu commented on FLINK-34214: --- Hi, may i consulting one question that if we disable the push down for JDBC/MongoDB source, does it mean the source will scan whole data? The performance of scan whole data is worse than filter data via unindexed column? > FLIP-377: Support fine-grained configuration to control filter push down for > Table/SQL Sources > -- > > Key: FLINK-34214 > URL: https://issues.apache.org/jira/browse/FLINK-34214 > Project: Flink > Issue Type: New Feature > Components: Connectors / JDBC, Connectors / MongoDB >Affects Versions: mongodb-1.0.2, jdbc-3.1.2 >Reporter: Jiabao Sun >Priority: Major > Fix For: mongodb-1.1.0, jdbc-3.1.3 > > > This improvement implements [FLIP-377 Support fine-grained configuration to > control filter push down for Table/SQL > Sources|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768] > This FLIP has 2 goals: > * Introduces a new configuration filter.handling.policy to the JDBC and > MongoDB connector. > * Suggests a convention option name if other connectors are going to add an > option for the same purpose. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33263][bugfix][table-planner] Update the flink shaded guava version used to 32 [flink]
BIOINSu opened a new pull request, #24200: URL: https://github.com/apache/flink/pull/24200 ## What is the purpose of the change Bugfix for FLINK-33263. Update the flink shaded guava version used in `SourceTransformationWrapper` to 32. ## Brief change log - Update the flink shaded guava version used in `SourceTransformationWrapper` to 32. ## Verifying this change This change added tests and can be verified through CI progress. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [Draft][configuration] Change standard YAML parser to support YAML 1.2. [flink]
flinkbot commented on PR #24199: URL: https://github.com/apache/flink/pull/24199#issuecomment-1911560589 ## CI report: * a5f6ccf25c195d05cc1f64e59284c09b413dfccc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [Draft][configuration] Change standard YAML parser to support YAML 1.2. [flink]
JunRuiLee opened a new pull request, #24199: URL: https://github.com/apache/flink/pull/24199 ## What is the purpose of the change Change standard YAML parser to support YAML 1.2. ## Brief change log - Change snakeyaml to snakeyaml engine 2.6 to support YAML 1.2 - Change pyyaml to ruamel.yaml 0.18.4 to support YAML 1.2 ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (**yes** / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34145][connector/filesystem] support dynamic source parallelism inference in batch jobs [flink]
SinBex commented on PR #24186: URL: https://github.com/apache/flink/pull/24186#issuecomment-1911552066 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34222][table] End-to-end implementation of minibatch join [flink]
xishuaidelin commented on code in PR #24161:
URL: https://github.com/apache/flink/pull/24161#discussion_r1467258035
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalJoin.scala:
##
@@ -112,6 +112,7 @@ class StreamPhysicalJoin(
getUpsertKeys(right, joinSpec.getRightKeys)
)
)
+ .itemIf("miniBatch", "true",
JoinUtil.isMiniBatchEnabled(unwrapTableConfig(this)))
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34222][table] End-to-end implementation of minibatch join [flink]
xishuaidelin commented on code in PR #24161:
URL: https://github.com/apache/flink/pull/24161#discussion_r1467257676
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/MinibatchUtil.java:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils;
+
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.operators.bundle.trigger.CoBundleTrigger;
+import
org.apache.flink.table.runtime.operators.bundle.trigger.CountBundleTrigger;
+import
org.apache.flink.table.runtime.operators.bundle.trigger.CountCoBundleTrigger;
+
+/** Utility class for mini-batch related config. */
+public class MinibatchUtil {
+
+/**
+ * Check if MiniBatch is enabled.
+ *
+ * @param config config
+ * @return true if MiniBatch enabled else false.
+ */
+public static boolean isMiniBatchEnabled(ReadableConfig config) {
+return config.get(ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED);
+}
+
+/**
+ * Creates a MiniBatch trigger depends on the config for one input.
+ *
+ * @param config config
+ * @return MiniBatch trigger
+ */
+public static CountBundleTrigger
createMiniBatchTrigger(ReadableConfig config) {
+long size = miniBatchSize(config);
+return new CountBundleTrigger<>(size);
+}
+
+/**
+ * Creates a MiniBatch trigger depends on the config for two input.
+ *
+ * @param config config
+ * @return MiniBatch trigger
+ */
+public static CoBundleTrigger createCoMiniBatchTrigger(
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34058][table] Support optional parameters for named parameters [flink]
xuyangzhong commented on PR #24183: URL: https://github.com/apache/flink/pull/24183#issuecomment-1911536124 BTW, can you update the description of this pr? Currently they seem to come with the template. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34240) The example of sliding windows with offset in documentation is not correct
Weijie Guo created FLINK-34240:
--
Summary: The example of sliding windows with offset in
documentation is not correct
Key: FLINK-34240
URL: https://issues.apache.org/jira/browse/FLINK-34240
Project: Flink
Issue Type: Bug
Components: Documentation
Reporter: Weijie Guo
Assignee: Weijie Guo
In documentation of windows, we have the following example code:
{code:java}
// sliding processing-time windows offset by -8 hours
input
.keyBy()
.window(SlidingProcessingTimeWindows.of(Time.hours(12), Time.hours(1),
Time.hours(-8)))
.();
{code}
Unfortunately, it will raise as the absolute value of offset must be less than
the slide.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-34058) Support optional parameters for named parameters
[ https://issues.apache.org/jira/browse/FLINK-34058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34058: --- Labels: pull-request-available (was: ) > Support optional parameters for named parameters > > > Key: FLINK-34058 > URL: https://issues.apache.org/jira/browse/FLINK-34058 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Feng Jin >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34058][table] Support optional parameters for named parameters [flink]
xuyangzhong commented on code in PR #24183:
URL: https://github.com/apache/flink/pull/24183#discussion_r1467163962
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/runtime/stream/sql/FunctionITCase.java:
##
@@ -1073,6 +1073,25 @@ void testNamedArgumentsTableFunction() throws Exception {
assertThat(TestCollectionTableFactory.getResult()).containsExactlyInAnyOrder(sinkData);
}
+@Test
+void testNamedArgumentsCatalogTableFunctionWithOptionalArguments() throws
Exception {
Review Comment:
nit: remove `Catalog`
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/extraction/FunctionTemplate.java:
##
@@ -194,14 +194,17 @@ private static T defaultAsNull(
"Argument and input hints cannot be declared in the same
function hint.");
}
+Boolean[] argumentOptionals = null;
Review Comment:
Just a little curious, will there be a null Boolean value as element?
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/calls/StringCallGen.scala:
##
@@ -237,6 +238,9 @@ object StringCallGen {
val currentDatabase = ctx.addReusableQueryLevelCurrentDatabase()
generateNonNullField(returnType, currentDatabase)
+ case DEFAULT =>
Review Comment:
The class `StringCallGen` contains all the functions that take `StringData`
as input or output parameters. It's not quite appropriate to place the function
`DEFAULT` here.
How about we directly put it into the
`ExprCodeGenerator#generateCallExpression` and add `public static final
SqlSpecialOperator DEFAULT = SqlStdOperatorTable.DEFAULT;` in
`FlinkSqlOperatorTable`?
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/extraction/FunctionSignatureTemplate.java:
##
@@ -67,7 +73,24 @@ static FunctionSignatureTemplate of(
throw extractionError(
"Argument name conflict, there are at least two argument
names that are the same.");
}
-return new FunctionSignatureTemplate(argumentTemplates, isVarArgs,
argumentNames);
+if (argumentOptionals != null && argumentOptionals.length !=
argumentTemplates.size()) {
+throw extractionError(
+"Mismatch between number of argument optionals '%s' and
argument types '%s'.",
+argumentOptionals.length, argumentTemplates.size());
+}
+if (argumentOptionals != null) {
+for (int i = 0; i < argumentTemplates.size(); i++) {
+DataType dataType = argumentTemplates.get(i).dataType;
+if (dataType != null
+&& !dataType.getLogicalType().isNullable()
Review Comment:
Nice validation!
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/TypeInferenceOperandChecker.java:
##
@@ -116,7 +127,17 @@ public Consistency getConsistency() {
@Override
public boolean isOptional(int i) {
-return false;
+Optional> optionalArguments =
typeInference.getOptionalArguments();
+if (optionalArguments.isPresent()) {
+return optionalArguments.get().get(i);
Review Comment:
Nit: Will a NullPointerException occur here when converting from Boolean to
boolean?
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/inference/TypeInferenceOperandChecker.java:
##
@@ -116,7 +127,17 @@ public Consistency getConsistency() {
@Override
public boolean isOptional(int i) {
-return false;
+Optional> optionalArguments =
typeInference.getOptionalArguments();
+if (optionalArguments.isPresent()) {
+return optionalArguments.get().get(i);
+} else {
+return false;
+}
+}
+
+@Override
+public boolean isFixedParameters() {
+return typeInference.getTypedArguments().isPresent();
Review Comment:
As far as the current situation is concerned, this code is also fine. (I'm
wondering if it's possible to use the `isVarArg` on `FunctionHint`? )
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/converters/SqlProcedureCallConverter.java:
##
@@ -70,13 +73,30 @@ public Operation convertSqlNode(SqlNode sqlNode,
ConvertContext context) {
SqlCallBinding sqlCallBinding =
new SqlCallBinding(context.getSqlValidator(), null,
callProcedure);
+
+TypeInference typeInference =
+procedureDefinition.getTypeInference(
+context.getCatalogManager().getDataTypeFactory());
+List reducedOperands =
reduceOperands(sqlCallBinding.operands(), context);
Review Comment:
If we union the logic about correcting the return type of `DEFAULT`, are
these codes in
Re: [PR] [FLINK-33899][connectors/mongodb] Java 17 and 21 support for mongodb connector [flink-connector-mongodb]
Jiabao-Sun commented on code in PR #21: URL: https://github.com/apache/flink-connector-mongodb/pull/21#discussion_r1467234659 ## .github/workflows/push_pr.yml: ## @@ -25,7 +25,14 @@ jobs: compile_and_test: strategy: matrix: -flink: [1.16-SNAPSHOT, 1.17-SNAPSHOT, 1.18-SNAPSHOT, 1.19-SNAPSHOT] +flink: [ 1.16-SNAPSHOT, 1.17-SNAPSHOT ] +jdk: [ '8, 11' ] +include: + - flink: 1.18-SNAPSHOT +jdk: '8, 11, 17' + - flink: 1.19-SNAPSHOT +jdk: '8, 11, 17, 21' Review Comment: This modification refers to the jdbc-connector. https://github.com/apache/flink-connector-jdbc/blob/e3dd84160cd665ae17672da8b6e742e61a72a32d/.github/workflows/push_pr.yml#L19-L38 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34237][connectors/mongodb] Fix compile failed with Flink 1.19-SNAPSHOT [flink-connector-mongodb]
WencongLiu commented on PR #22: URL: https://github.com/apache/flink-connector-mongodb/pull/22#issuecomment-1911458296 Hi @Jiabao-Sun , I've opened another pull request to revert the breaking changes in flink repo. All connectors won't meet this issue after the PR is merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34237][connectors/mongodb] Fix compile failed with Flink 1.19-SNAPSHOT [flink-connector-mongodb]
Jiabao-Sun commented on PR #22: URL: https://github.com/apache/flink-connector-mongodb/pull/22#issuecomment-1911448586 The modification to MongoWriterITCase is due to https://github.com/apache/flink/pull/24180#discussion_r1467184292. @leonardBang please help take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34059][docs][table] Add documentation on STATE_TTL hint [flink]
flinkbot commented on PR #24198: URL: https://github.com/apache/flink/pull/24198#issuecomment-1911441439 ## CI report: * 3a87f3409f768c3e1eead3f281e6d326b48b7d89 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34209] Migrate FileSink to the new SinkV2 API [flink]
Jiabao-Sun commented on code in PR #24180:
URL: https://github.com/apache/flink/pull/24180#discussion_r1467200868
##
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/sink/writer/TestSinkInitContext.java:
##
@@ -48,7 +48,7 @@
import java.util.concurrent.ScheduledFuture;
/** A mock implementation of a {@code Sink.InitContext} to be used in sink
unit tests. */
-public class TestSinkInitContext implements Sink.InitContext {
+public class TestSinkInitContext implements WriterInitContext {
Review Comment:
```java
@Deprecated
SinkWriter createWriter(InitContext context) throws IOException;
default SinkWriter createWriter(WriterInitContext context)
throws IOException {
return createWriter(new InitContextWrapper(context));
}
```
```java
TestSinkInitContext initContext = new TestSinkInitContext();
SinkWriter writer = sink.createWriter(initContext);
```
One solution is to not directly construct a Writer through
TestSinkInitContext, but instead use Sink.createWriter() to create a Writer.
So there is no need to revert it change.
Sorry for the disturbance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-34222) Get minibatch join operator involved
[ https://issues.apache.org/jira/browse/FLINK-34222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34222: --- Labels: pull-request-available (was: ) > Get minibatch join operator involved > > > Key: FLINK-34222 > URL: https://issues.apache.org/jira/browse/FLINK-34222 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Shuai Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Get minibatch join operator involved in which includes both plan and > operator. Implement minibatch join in E2E. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34053][table-planner] Support state ttl hint for group aggregate [flink]
xuyangzhong commented on code in PR #24179:
URL: https://github.com/apache/flink/pull/24179#discussion_r1466284522
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/optimize/QueryHintsResolver.java:
##
@@ -83,21 +85,50 @@ final List resolve(List roots) {
}
@Override
-protected RelNode visitBiRel(BiRel biRel) {
-Optional leftName = extractAliasOrTableName(biRel.getLeft());
-Optional rightName = extractAliasOrTableName(biRel.getRight());
-
-Set existentKVHints = new HashSet<>();
-
-List oldHints = ((Hintable) biRel).getHints();
+protected RelNode doVisit(RelNode node) {
+List oldHints = ((Hintable) node).getHints();
List oldQueryHints = FlinkHints.getAllQueryHints(oldHints);
// has no hints, return directly.
if (oldQueryHints.isEmpty()) {
-return super.visitChildren(biRel);
+return super.visitChildren(node);
}
-List newHints = new ArrayList<>();
+final List newHints;
+if (node instanceof BiRel) {
+BiRel biRel = (BiRel) node;
+Optional leftName =
extractAliasOrTableName(biRel.getLeft());
+Optional rightName =
extractAliasOrTableName(biRel.getRight());
+newHints = validateAndGetNewHintsFromBiRel(leftName, rightName,
oldHints);
Review Comment:
> Agree with you. Since the improvement work for the input type Optional is
not a part of this pr, I have created a new jira for it.
https://issues.apache.org/jira/browse/FLINK-34235
Assigned to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34222][table] End-to-end implementation of minibatch join [flink]
lsyldliu commented on code in PR #24161:
URL: https://github.com/apache/flink/pull/24161#discussion_r1467198322
##
flink-table/flink-table-planner/src/main/scala/org/apache/flink/table/planner/plan/nodes/physical/stream/StreamPhysicalJoin.scala:
##
@@ -112,6 +112,7 @@ class StreamPhysicalJoin(
getUpsertKeys(right, joinSpec.getRightKeys)
)
)
+ .itemIf("miniBatch", "true",
JoinUtil.isMiniBatchEnabled(unwrapTableConfig(this)))
Review Comment:
```suggestion
.itemIf("miniBatch", "true",
MinibatchUtil.isMiniBatchEnabled(unwrapTableConfig(this)))
```
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/utils/MinibatchUtil.java:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.utils;
+
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.api.config.ExecutionConfigOptions;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.runtime.operators.bundle.trigger.CoBundleTrigger;
+import
org.apache.flink.table.runtime.operators.bundle.trigger.CountBundleTrigger;
+import
org.apache.flink.table.runtime.operators.bundle.trigger.CountCoBundleTrigger;
+
+/** Utility class for mini-batch related config. */
+public class MinibatchUtil {
+
+/**
+ * Check if MiniBatch is enabled.
+ *
+ * @param config config
+ * @return true if MiniBatch enabled else false.
+ */
+public static boolean isMiniBatchEnabled(ReadableConfig config) {
+return config.get(ExecutionConfigOptions.TABLE_EXEC_MINIBATCH_ENABLED);
+}
+
+/**
+ * Creates a MiniBatch trigger depends on the config for one input.
+ *
+ * @param config config
+ * @return MiniBatch trigger
+ */
+public static CountBundleTrigger
createMiniBatchTrigger(ReadableConfig config) {
+long size = miniBatchSize(config);
+return new CountBundleTrigger<>(size);
+}
+
+/**
+ * Creates a MiniBatch trigger depends on the config for two input.
+ *
+ * @param config config
+ * @return MiniBatch trigger
+ */
+public static CoBundleTrigger createCoMiniBatchTrigger(
Review Comment:
createMiniBatchCoTrigger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33682][metrics] Reuse source operator input records/bytes metrics for SourceOperatorStreamTask [flink]
X-czh closed pull request #23823: [FLINK-33682][metrics] Reuse source operator input records/bytes metrics for SourceOperatorStreamTask URL: https://github.com/apache/flink/pull/23823 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34053][table-planner] Support state ttl hint for group aggregate [flink]
LadyForest commented on code in PR #24179:
URL: https://github.com/apache/flink/pull/24179#discussion_r1467200085
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/optimize/QueryHintsResolver.java:
##
@@ -83,21 +85,50 @@ final List resolve(List roots) {
}
@Override
-protected RelNode visitBiRel(BiRel biRel) {
-Optional leftName = extractAliasOrTableName(biRel.getLeft());
-Optional rightName = extractAliasOrTableName(biRel.getRight());
-
-Set existentKVHints = new HashSet<>();
-
-List oldHints = ((Hintable) biRel).getHints();
+protected RelNode doVisit(RelNode node) {
+List oldHints = ((Hintable) node).getHints();
List oldQueryHints = FlinkHints.getAllQueryHints(oldHints);
// has no hints, return directly.
if (oldQueryHints.isEmpty()) {
-return super.visitChildren(biRel);
+return super.visitChildren(node);
}
-List newHints = new ArrayList<>();
+final List newHints;
+if (node instanceof BiRel) {
+BiRel biRel = (BiRel) node;
+Optional leftName =
extractAliasOrTableName(biRel.getLeft());
+Optional rightName =
extractAliasOrTableName(biRel.getRight());
+newHints = validateAndGetNewHintsFromBiRel(leftName, rightName,
oldHints);
Review Comment:
> https://issues.apache.org/jira/browse/FLINK-34235
Assigned to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-34235) Not using Optional as input arguments in QueryHintsResolver
[ https://issues.apache.org/jira/browse/FLINK-34235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jane Chan reassigned FLINK-34235: - Assignee: xuyang > Not using Optional as input arguments in QueryHintsResolver > --- > > Key: FLINK-34235 > URL: https://issues.apache.org/jira/browse/FLINK-34235 > Project: Flink > Issue Type: Technical Debt > Components: Table SQL / Planner >Reporter: xuyang >Assignee: xuyang >Priority: Not a Priority > > It also seems that we always can't get an empty left or right name from input. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34115) TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate fails
[
https://issues.apache.org/jira/browse/FLINK-34115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1787#comment-1787
]
Jane Chan commented on FLINK-34115:
---
{quote}1.18 build that didn't include the fix, yet:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56793=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11745]
{quote}
I think this build does not include 0a1d671d0c7c5912de2116dbf1d1d641cff72b95.
> TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate fails
> --
>
> Key: FLINK-34115
> URL: https://issues.apache.org/jira/browse/FLINK-34115
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Matthias Pohl
>Assignee: Jane Chan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0, 1.18.2
>
>
> It failed twice in the same pipeline run:
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56348=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11613]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56348=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11963]
> {code:java}
> Jan 14 01:20:01 01:20:01.949 [ERROR] Tests run: 18, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 29.07 s <<< FAILURE! -- in
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase
> Jan 14 01:20:01 01:20:01.949 [ERROR]
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate
> -- Time elapsed: 0.518 s <<< FAILURE!
> Jan 14 01:20:01 org.opentest4j.AssertionFailedError:
> Jan 14 01:20:01
> Jan 14 01:20:01 expected: List((true,6,1), (false,6,1), (true,6,1),
> (true,3,2), (false,6,1), (false,3,2), (true,6,1), (true,5,2), (false,6,1),
> (false,5,2), (true,8,1), (true,6,2), (false,8,1), (false,6,2), (true,8,1),
> (true,6,2))
> Jan 14 01:20:01 but was: List((true,3,1), (false,3,1), (true,5,1),
> (true,3,2), (false,5,1), (false,3,2), (true,8,1), (true,5,2), (false,8,1),
> (false,5,2), (true,8,1), (true,5,2), (false,8,1), (false,5,2), (true,8,1),
> (true,6,2))
> Jan 14 01:20:01 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Jan 14 01:20:01 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Jan 14 01:20:01 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Jan 14 01:20:01 at
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase.checkRank$1(TableAggregateITCase.scala:122)
> Jan 14 01:20:01 at
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate(TableAggregateITCase.scala:69)
> Jan 14 01:20:01 at java.lang.reflect.Method.invoke(Method.java:498)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Jan 14 01:20:01 at
> scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:26)
> Jan 14 01:20:01 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Jan 14 01:20:01 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Jan 14 01:20:01 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Jan 14 01:20:01 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> Jan 14 01:20:01 at
>
[jira] [Commented] (FLINK-34229) Duplicate entry in InnerClasses attribute in class file FusionStreamOperator
[
https://issues.apache.org/jira/browse/FLINK-34229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1785#comment-1785
]
Dan Zou commented on FLINK-34229:
-
[~xiasun], I could not reproduce in my environment, could you please turn on
the debug log and obtain the generated code in the log in CompileUtils#
doCompile.
{code:java}
private static Class doCompile(ClassLoader cl, String name, String
code) {
checkNotNull(cl, "Classloader must not be null.");
CODE_LOG.debug("Compiling: {} \n\n Code:\n{}", name, code)
...
}
{code}
> Duplicate entry in InnerClasses attribute in class file FusionStreamOperator
>
>
> Key: FLINK-34229
> URL: https://issues.apache.org/jira/browse/FLINK-34229
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.19.0
>Reporter: xingbe
>Priority: Major
> Attachments: image-2024-01-24-17-05-47-883.png
>
>
> I noticed a runtime error happens in 10TB TPC-DS (q35.sql) benchmarks in
> 1.19, the problem did not happen in 1.18.0. This issue may have been newly
> introduced recently. !image-2024-01-24-17-05-47-883.png|width=589,height=279!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-34237] Revert the breaking changes to SourceReaderContext [flink]
flinkbot commented on PR #24197: URL: https://github.com/apache/flink/pull/24197#issuecomment-1911434979 ## CI report: * 486d33707572c3b6cfd2f286d34d9c3648ba14ed UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-34059) Add documentation on how to use state TTL hint
[ https://issues.apache.org/jira/browse/FLINK-34059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34059: --- Labels: pull-request-available (was: ) > Add documentation on how to use state TTL hint > -- > > Key: FLINK-34059 > URL: https://issues.apache.org/jira/browse/FLINK-34059 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Table SQL / API >Affects Versions: 1.19.0 >Reporter: Jane Chan >Assignee: Jane Chan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-34059][docs][table] Add documentation on STATE_TTL hint [flink]
LadyForest opened a new pull request, #24198: URL: https://github.com/apache/flink/pull/24198 ## What is the purpose of the change This PR adds a doc to the `STATE_TTL` hint. ## Brief change log Add doc. ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): No - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: No - The serializers: No - The runtime per-record code paths (performance sensitive): No - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: No - The S3 file system connector: No ## Documentation - Does this pull request introduce a new feature? Yes - If yes, how is the feature documented? docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34115) TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate fails
[
https://issues.apache.org/jira/browse/FLINK-34115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1782#comment-1782
]
Jane Chan commented on FLINK-34115:
---
Hi [~mapohl], Thank you for pointing out that the bug fix did not work as
expected. I apologize for any inconvenience. I'll revisit the bug report to
reproduce the issue before submitting any further bug fixes. I appreciate your
patience in resolving this issue.
> TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate fails
> --
>
> Key: FLINK-34115
> URL: https://issues.apache.org/jira/browse/FLINK-34115
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Matthias Pohl
>Assignee: Jane Chan
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0, 1.18.2
>
>
> It failed twice in the same pipeline run:
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56348=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11613]
> *
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56348=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11963]
> {code:java}
> Jan 14 01:20:01 01:20:01.949 [ERROR] Tests run: 18, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 29.07 s <<< FAILURE! -- in
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase
> Jan 14 01:20:01 01:20:01.949 [ERROR]
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate
> -- Time elapsed: 0.518 s <<< FAILURE!
> Jan 14 01:20:01 org.opentest4j.AssertionFailedError:
> Jan 14 01:20:01
> Jan 14 01:20:01 expected: List((true,6,1), (false,6,1), (true,6,1),
> (true,3,2), (false,6,1), (false,3,2), (true,6,1), (true,5,2), (false,6,1),
> (false,5,2), (true,8,1), (true,6,2), (false,8,1), (false,6,2), (true,8,1),
> (true,6,2))
> Jan 14 01:20:01 but was: List((true,3,1), (false,3,1), (true,5,1),
> (true,3,2), (false,5,1), (false,3,2), (true,8,1), (true,5,2), (false,8,1),
> (false,5,2), (true,8,1), (true,5,2), (false,8,1), (false,5,2), (true,8,1),
> (true,6,2))
> Jan 14 01:20:01 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Jan 14 01:20:01 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Jan 14 01:20:01 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Jan 14 01:20:01 at
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase.checkRank$1(TableAggregateITCase.scala:122)
> Jan 14 01:20:01 at
> org.apache.flink.table.planner.runtime.stream.table.TableAggregateITCase.testFlagAggregateWithOrWithoutIncrementalUpdate(TableAggregateITCase.scala:69)
> Jan 14 01:20:01 at java.lang.reflect.Method.invoke(Method.java:498)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> Jan 14 01:20:01 at
> java.util.Iterator.forEachRemaining(Iterator.java:116)
> Jan 14 01:20:01 at
> scala.collection.convert.Wrappers$IteratorWrapper.forEachRemaining(Wrappers.scala:26)
> Jan 14 01:20:01 at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> Jan 14 01:20:01 at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> Jan 14 01:20:01 at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Jan 14 01:20:01 at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Jan 14 01:20:01 at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Jan 14 01:20:01 at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
> Jan 14 01:20:01 at
>
[jira] [Updated] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-34237:
---
Labels: pull-request-available (was: )
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Assignee: Wencong Liu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19.
> Another point: The FLIP-382 didn't mentioned the connector backward
> compatibility well, maybe we need to rethink the section. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-34237] Revert the breaking changes to SourceReaderContext [flink]
WencongLiu opened a new pull request, #24197: URL: https://github.com/apache/flink/pull/24197 ## What is the purpose of the change *Revert the breaking changes to SourceReaderContext.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34200) AutoRescalingITCase#testCheckpointRescalingInKeyedState fails
[
https://issues.apache.org/jira/browse/FLINK-34200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811109#comment-17811109
]
Rui Fan commented on FLINK-34200:
-
Thanks [~mapohl] for adding more failed cases.
As [~srichter] feedback in FLINK-34218, the reason of test fails isn't test
code. So I guess current master branch might has some potential bugs related to
correctness. I'm not sure should we mark this bug as the Blocker or Critical?
> AutoRescalingITCase#testCheckpointRescalingInKeyedState fails
> -
>
> Key: FLINK-34200
> URL: https://issues.apache.org/jira/browse/FLINK-34200
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56601=logs=8fd9202e-fd17-5b26-353c-ac1ff76c8f28=ea7cf968-e585-52cb-e0fc-f48de023a7ca=8200]
> {code:java}
> Jan 19 02:31:53 02:31:53.954 [ERROR] Tests run: 32, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 1050 s <<< FAILURE! -- in
> org.apache.flink.test.checkpointing.AutoRescalingITCase
> Jan 19 02:31:53 02:31:53.954 [ERROR]
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState[backend
> = rocksdb, buffersPerChannel = 2] -- Time elapsed: 59.10 s <<< FAILURE!
> Jan 19 02:31:53 java.lang.AssertionError: expected:<[(0,8000), (0,32000),
> (0,48000), (0,72000), (1,78000), (1,3), (1,54000), (0,2000), (0,1),
> (0,5), (0,66000), (0,74000), (0,82000), (1,8), (1,0), (1,16000),
> (1,24000), (1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000),
> (0,6), (0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000),
> (1,58000), (0,6000), (0,14000), (0,22000), (0,38000), (0,46000), (0,62000),
> (0,7), (1,4000), (1,2), (1,36000), (1,44000)]> but was:<[(0,8000),
> (0,32000), (0,48000), (0,72000), (1,78000), (1,3), (1,54000), (0,2000),
> (0,1), (0,5), (0,66000), (0,74000), (0,82000), (1,8), (1,0),
> (1,16000), (1,24000), (1,4), (1,56000), (1,64000), (0,12000), (0,28000),
> (0,52000), (0,6), (0,68000), (0,76000), (0,1000), (0,25000), (0,33000),
> (0,41000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000), (0,6000),
> (0,14000), (0,22000), (0,38000), (0,46000), (0,62000), (0,7), (1,4000),
> (1,2), (1,36000), (1,44000)]>
> Jan 19 02:31:53 at org.junit.Assert.fail(Assert.java:89)
> Jan 19 02:31:53 at org.junit.Assert.failNotEquals(Assert.java:835)
> Jan 19 02:31:53 at org.junit.Assert.assertEquals(Assert.java:120)
> Jan 19 02:31:53 at org.junit.Assert.assertEquals(Assert.java:146)
> Jan 19 02:31:53 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingKeyedState(AutoRescalingITCase.java:296)
> Jan 19 02:31:53 at
> org.apache.flink.test.checkpointing.AutoRescalingITCase.testCheckpointRescalingInKeyedState(AutoRescalingITCase.java:196)
> Jan 19 02:31:53 at java.lang.reflect.Method.invoke(Method.java:498)
> Jan 19 02:31:53 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33263][table-planner] Implement ParallelismProvider for sources in the table planner [flink]
libenchao commented on code in PR #24128: URL: https://github.com/apache/flink/pull/24128#discussion_r1467192709 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SourceTransformationWrapper.java: ## @@ -0,0 +1,72 @@ +/* +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package org.apache.flink.streaming.api.transformations; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.dag.Transformation; +import org.apache.flink.streaming.api.graph.TransformationTranslator; + +import org.apache.flink.shaded.guava31.com.google.common.collect.Lists; Review Comment: I'll send a notice to the dev@ to let our developers know it in case of confusions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-34239) Introduce a deep copy method of SerializerConfig for merging with Table configs in org.apache.flink.table.catalog.DataTypeFactoryImpl
Zhanghao Chen created FLINK-34239: - Summary: Introduce a deep copy method of SerializerConfig for merging with Table configs in org.apache.flink.table.catalog.DataTypeFactoryImpl Key: FLINK-34239 URL: https://issues.apache.org/jira/browse/FLINK-34239 Project: Flink Issue Type: Sub-task Components: API / Core Affects Versions: 1.19.0 Reporter: Zhanghao Chen *Problem* Currently, org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig will create a deep-copy of the SerializerConfig and merge Table config into it. However, the deep copy is done by manully calling the getter and setter methods of SerializerConfig, and is prone to human errors, e.g. missing copying a newly added field in SerializerConfig. *Proposal* Introduce a deep copy method for SerializerConfig and replace the curr impl in org.apache.flink.table.catalog.DataTypeFactoryImpl#createSerializerExecutionConfig. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33263][table-planner] Implement ParallelismProvider for sources in the table planner [flink]
libenchao commented on code in PR #24128: URL: https://github.com/apache/flink/pull/24128#discussion_r1467192424 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SourceTransformationWrapper.java: ## @@ -0,0 +1,72 @@ +/* +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package org.apache.flink.streaming.api.transformations; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.dag.Transformation; +import org.apache.flink.streaming.api.graph.TransformationTranslator; + +import org.apache.flink.shaded.guava31.com.google.common.collect.Lists; Review Comment: @RocMarshal Thanks for spotting it, we've noticed it, and @X-czh will fix this using a hotfix commit in https://github.com/apache/flink/pull/24133. Hopefully we can merge it after the CI passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33263][table-planner] Implement ParallelismProvider for sources in the table planner [flink]
RocMarshal commented on code in PR #24128: URL: https://github.com/apache/flink/pull/24128#discussion_r1467191595 ## flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SourceTransformationWrapper.java: ## @@ -0,0 +1,72 @@ +/* +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package org.apache.flink.streaming.api.transformations; + +import org.apache.flink.annotation.Internal; +import org.apache.flink.api.common.ExecutionConfig; +import org.apache.flink.api.dag.Transformation; +import org.apache.flink.streaming.api.graph.TransformationTranslator; + +import org.apache.flink.shaded.guava31.com.google.common.collect.Lists; Review Comment: hi @BIOINSu thx for the contribution. Would the `guava31` be `guava32` ? CC @libenchao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-33465) Make SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving.
[
https://issues.apache.org/jira/browse/FLINK-33465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren closed FLINK-33465.
-
Resolution: Duplicate
> Make SingleThreadFetcherManager and FutureCompletingBlockingQueue as
> PublicEvolving.
>
>
> Key: FLINK-33465
> URL: https://issues.apache.org/jira/browse/FLINK-33465
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Parent
>Affects Versions: 1.18.0
>Reporter: Hongshun Wang
>Priority: Major
> Fix For: 1.19.0
>
>
> As discussed in FLINK-31324, though the {{SingleThreadFetcherManager}} is
> annotated as {{{}Internal{}}}, it actually acts as some-degree public API,
> which is widely used in many connector projects:
> [flink-cdc-connector|https://github.com/ververica/flink-cdc-connectors/blob/release-2.3.0/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/reader/MySqlSourceReader.java#L93],
>
> [flink-connector-mongodb|https://github.com/apache/flink-connector-mongodb/blob/main/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReader.java#L58]
> and so on.
> More over, even the constructor of
> `SingleThreadMultiplexSourceReaderBase` (which is PublicEvolving) includes
> the params of `SingleThreadFetcherManager` and
> `FutureCompletingBlockingQueue` .That means that the
> `SingleThreadFetcherManager` and
> `FutureCompletingBlockingQueue`have already been exposed to users for a long
> time and are widely used.
>
> {code:java}
> public SingleThreadMultiplexSourceReaderBase(
> FutureCompletingBlockingQueue> elementsQueue,
> SingleThreadFetcherManager splitFetcherManager,
> RecordEmitter recordEmitter,
> Configuration config,
> SourceReaderContext context)
> { super(elementsQueue, splitFetcherManager, recordEmitter, config, context); }
>
> {code}
>
> Therefore, why not make SingleThreadFetcherManager and
> FutureCompletingBlockingQueue PublicEvolving?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33465) Make SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving.
[
https://issues.apache.org/jira/browse/FLINK-33465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren updated FLINK-33465:
--
Fix Version/s: (was: 1.19.0)
> Make SingleThreadFetcherManager and FutureCompletingBlockingQueue as
> PublicEvolving.
>
>
> Key: FLINK-33465
> URL: https://issues.apache.org/jira/browse/FLINK-33465
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Parent
>Affects Versions: 1.18.0
>Reporter: Hongshun Wang
>Priority: Major
>
> As discussed in FLINK-31324, though the {{SingleThreadFetcherManager}} is
> annotated as {{{}Internal{}}}, it actually acts as some-degree public API,
> which is widely used in many connector projects:
> [flink-cdc-connector|https://github.com/ververica/flink-cdc-connectors/blob/release-2.3.0/flink-connector-mysql-cdc/src/main/java/com/ververica/cdc/connectors/mysql/source/reader/MySqlSourceReader.java#L93],
>
> [flink-connector-mongodb|https://github.com/apache/flink-connector-mongodb/blob/main/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReader.java#L58]
> and so on.
> More over, even the constructor of
> `SingleThreadMultiplexSourceReaderBase` (which is PublicEvolving) includes
> the params of `SingleThreadFetcherManager` and
> `FutureCompletingBlockingQueue` .That means that the
> `SingleThreadFetcherManager` and
> `FutureCompletingBlockingQueue`have already been exposed to users for a long
> time and are widely used.
>
> {code:java}
> public SingleThreadMultiplexSourceReaderBase(
> FutureCompletingBlockingQueue> elementsQueue,
> SingleThreadFetcherManager splitFetcherManager,
> RecordEmitter recordEmitter,
> Configuration config,
> SourceReaderContext context)
> { super(elementsQueue, splitFetcherManager, recordEmitter, config, context); }
>
> {code}
>
> Therefore, why not make SingleThreadFetcherManager and
> FutureCompletingBlockingQueue PublicEvolving?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34230) Update flink-docs README: add -Pskip-webui-build to the config doc generation command
[ https://issues.apache.org/jira/browse/FLINK-34230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811104#comment-17811104 ] Zhanghao Chen commented on FLINK-34230: --- [~Weijie Guo] Could you help take a look when you are free? A simple doc improvement, discovered when I was modifying config options frequently. > Update flink-docs README: add -Pskip-webui-build to the config doc generation > command > - > > Key: FLINK-34230 > URL: https://issues.apache.org/jira/browse/FLINK-34230 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Zhanghao Chen >Priority: Minor > Labels: pull-request-available > > I used to generate Flink config docs with the command listed in flink-docs > README: > "mvn package -Dgenerate-config-docs -pl flink-docs -am -nsu DskipTests". The > command will compile Flink web frontend, which usu. takes a few minutes and > is in fact unnecessary. We can add "-Pskip-webui-build" to the config doc > generation command to save us a few minutes each time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811103#comment-17811103
]
Leonard Xu commented on FLINK-34237:
[~martijnvisser] ,this issue is different with FLINK-34149, this is introduced
by FLIP-382 and we should fix it as [~xtsong] commented
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Assignee: Wencong Liu
>Priority: Blocker
> Fix For: 1.19.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19.
> Another point: The FLIP-382 didn't mentioned the connector backward
> compatibility well, maybe we need to rethink the section. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-25509][connector/kafka] Add RecordEvaluator to dynamically stop source based on de-serialized records [flink-connector-kafka]
ruanhang1993 commented on code in PR #76:
URL:
https://github.com/apache/flink-connector-kafka/pull/76#discussion_r1467189983
##
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java:
##
@@ -350,6 +352,26 @@ public KafkaSourceBuilder setValueOnlyDeserializer(
return this;
}
+/**
+ * Sets the optional {@link RecordEvaluator eofRecordEvaluator} for
KafkaSource.
+ *
+ * When the evaluator is specified, it is invoked for each
de-serialized record to determine
+ * whether the corresponding split has reached end of stream. If a record
is matched by the
+ * evaluator, the source would not emit this record as well as the
following records in the same
+ * split.
+ *
+ * Note that the evaluator works jointly with the stopping offsets
specified by the {@link
+ * #setBounded(OffsetsInitializer)} or the {@link
#setUnbounded(OffsetsInitializer)}. The source
+ * stops consuming from a split when any of these conditions is met.
+ *
+ * @param eofRecordEvaluator a {@link RecordEvaluator recordEvaluator}
+ * @return this KafkaSourceBuilder.
+ */
+public KafkaSourceBuilder setEofRecordEvaluator(RecordEvaluator
eofRecordEvaluator) {
Review Comment:
Thanks for this comment, @mas-chen .
Yes, we should make the source FINISHED. I will fix this part later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-34196) FLIP-389 Annotate SingleThreadFetcherManager as PublicEvolving.
[ https://issues.apache.org/jira/browse/FLINK-34196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811102#comment-17811102 ] Qingsheng Ren commented on FLINK-34196: --- Merged to master: 6afee1de6585074e0df6205f1f52bb239dcf4a77 > FLIP-389 Annotate SingleThreadFetcherManager as PublicEvolving. > --- > > Key: FLINK-34196 > URL: https://issues.apache.org/jira/browse/FLINK-34196 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.18.1 >Reporter: Hongshun Wang >Assignee: Hongshun Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > This improvement implements > [FLIP-389](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465498) > This flip has 2 goals: > * To expose the SplitFetcherManager / SingleThreadFetcheManager as Public, > allowing connector developers to easily create their own threading models in > the SourceReaderBase. > * To hide the element queue from the connector developers and make > SplitFetcherManager the only owner class of the queue -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-34237:
---
Description:
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile)
on project flink-connector-mongodb: Compilation failure
134Error:
/home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
not abstract and does not override abstract method getTaskInfo() in
org.apache.flink.api.connector.source.SourceReaderContext
135{code}
[https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
This is related to FLINK-33905
One point: As
[FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
is accepted, all connectors who implement SourceReaderContext (i.e
MongoSourceReaderContext) should implement new introduced methods `
getTaskInfo()` if they want to compile/work with Flink 1.19.
Another point: The FLIP-382 didn't mentioned the connector backward
compatibility well, maybe we need to rethink the section. As I just have a
rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment under
this issue.
was:
{code:java}
Error: Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile)
on project flink-connector-mongodb: Compilation failure
134Error:
/home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
not abstract and does not override abstract method getTaskInfo() in
org.apache.flink.api.connector.source.SourceReaderContext
135{code}
[https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
This is related to FLINK-33905
One point: As
[FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
is accepted, all connectors who implement SourceReaderContext (i.e
MongoSourceReaderContext) should implement new introduced methods `
getTaskInfo()` if they want to compile/work with Flink 1.19. As I just have a
rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment under
this issue.
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Assignee: Wencong Liu
>Priority: Blocker
> Fix For: 1.19.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19.
> Another point: The FLIP-382 didn't mentioned the connector backward
> compatibility well, maybe we need to rethink the section. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811101#comment-17811101
]
Wencong Liu commented on FLINK-34237:
-
Thanks for the reminder. I'll fix it as soon as possible.
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Assignee: Wencong Liu
>Priority: Blocker
> Fix For: 1.19.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19.
> Another point: The FLIP-382 didn't mentioned the connector backward
> compatibility well, maybe we need to rethink the section. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-34196) FLIP-389 Annotate SingleThreadFetcherManager as PublicEvolving.
[ https://issues.apache.org/jira/browse/FLINK-34196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren resolved FLINK-34196. --- Resolution: Fixed > FLIP-389 Annotate SingleThreadFetcherManager as PublicEvolving. > --- > > Key: FLINK-34196 > URL: https://issues.apache.org/jira/browse/FLINK-34196 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.18.1 >Reporter: Hongshun Wang >Assignee: Hongshun Wang >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > This improvement implements > [FLIP-389](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465498) > This flip has 2 goals: > * To expose the SplitFetcherManager / SingleThreadFetcheManager as Public, > allowing connector developers to easily create their own threading models in > the SourceReaderBase. > * To hide the element queue from the connector developers and make > SplitFetcherManager the only owner class of the queue -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34196][Connectors][FLIP-389] Annotate SingleThreadFetcherManager as PublicEvolving. [flink]
PatrickRen merged PR #24171: URL: https://github.com/apache/flink/pull/24171 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34209] Migrate FileSink to the new SinkV2 API [flink]
Jiabao-Sun commented on code in PR #24180:
URL: https://github.com/apache/flink/pull/24180#discussion_r1467184292
##
flink-connectors/flink-connector-base/src/test/java/org/apache/flink/connector/base/sink/writer/TestSinkInitContext.java:
##
@@ -48,7 +48,7 @@
import java.util.concurrent.ScheduledFuture;
/** A mock implementation of a {@code Sink.InitContext} to be used in sink
unit tests. */
-public class TestSinkInitContext implements Sink.InitContext {
+public class TestSinkInitContext implements WriterInitContext {
Review Comment:
Hi @pvary, can we revert this change and provide a new class implements
`WriterInitContext`?
It may cause compatibility issues of the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34204][core] Dispatch legacy serializer related methods in ExecutionConfig and TypeInformation to SerializerConfig [flink]
reswqa commented on PR #24187: URL: https://github.com/apache/flink/pull/24187#issuecomment-1911329138 @X-czh Thanks for the review! I have removed the unused option in `test_controller.sh` and squashed all commits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34204][core] Dispatch legacy serializer related methods in ExecutionConfig and TypeInformation to SerializerConfig [flink]
reswqa commented on code in PR #24187: URL: https://github.com/apache/flink/pull/24187#discussion_r1467177600 ## tools/ci/test_controller.sh: ## @@ -105,7 +105,7 @@ if [ $STAGE == $STAGE_PYTHON ]; then run_with_watchdog "./flink-python/dev/lint-python.sh" $CALLBACK_ON_TIMEOUT EXIT_CODE=$? else - MVN_TEST_OPTIONS="-Dflink.tests.with-openssl -Dflink.tests.check-segment-multiple-free -Darchunit.freeze.store.default.allowStoreUpdate=false -Dpekko.rpc.force-invocation-serialization" + MVN_TEST_OPTIONS="--fail-at-end -Dflink.tests.with-openssl -Dflink.tests.check-segment-multiple-free -Darchunit.freeze.store.default.allowStoreUpdate=false -Dpekko.rpc.force-invocation-serialization" Review Comment: Removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34007][k8s] Adds workaround that fixes the deadlock when renewing the leadership lease fails [flink]
wangyang0918 commented on code in PR #24132:
URL: https://github.com/apache/flink/pull/24132#discussion_r1467160375
##
flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesLeaderElectorITCase.java:
##
@@ -110,4 +114,67 @@ void testMultipleKubernetesLeaderElectors() throws
Exception {
kubernetesExtension.getFlinkKubeClient().deleteConfigMap(leaderConfigMapName).get();
}
}
+
+/**
+ * This test verifies that the {@link KubernetesLeaderElector} is able to
handle scenario where
+ * the lease cannot be renewed.
+ *
+ * See FLINK-34007 for further details.
+ */
+@Test
+void testLeaderElectorLifecycleManagement() throws Exception {
Review Comment:
Great test. We might need to delete the leader elector ConfigMap because
fabricio `LeaderElector` does not clean up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34007][k8s] Adds workaround that fixes the deadlock when renewing the leadership lease fails [flink]
wangyang0918 commented on code in PR #24132:
URL: https://github.com/apache/flink/pull/24132#discussion_r1467160375
##
flink-kubernetes/src/test/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesLeaderElectorITCase.java:
##
@@ -110,4 +114,67 @@ void testMultipleKubernetesLeaderElectors() throws
Exception {
kubernetesExtension.getFlinkKubeClient().deleteConfigMap(leaderConfigMapName).get();
}
}
+
+/**
+ * This test verifies that the {@link KubernetesLeaderElector} is able to
handle scenario where
+ * the lease cannot be renewed.
+ *
+ * See FLINK-34007 for further details.
+ */
+@Test
+void testLeaderElectorLifecycleManagement() throws Exception {
Review Comment:
Great test. We might need to delete the leader elector ConfigMap because
fabricio {{LeaderElector}} does not clean up.
##
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/kubeclient/resources/KubernetesLeaderElector.java:
##
@@ -86,12 +117,38 @@ public KubernetesLeaderElector(
newLeader,
leaderConfig.getConfigMapName(
.build();
+this.executorService = executorService;
+
+LOG.info(
+"Create KubernetesLeaderElector on lock {}.",
+leaderElectionConfig.getLock().describe());
+}
+
+@GuardedBy("lock")
+private void resetInternalLeaderElector() {
+stopLeaderElectionCycle();
+
internalLeaderElector =
new LeaderElector(kubernetesClient, leaderElectionConfig,
executorService);
+currentLeaderElectionSession = internalLeaderElector.start();
+
LOG.info(
-"Create KubernetesLeaderElector {} with lock identity {}.",
-leaderConfig.getConfigMapName(),
-leaderConfig.getLockIdentity());
+"Triggered leader election on lock {}.",
leaderElectionConfig.getLock().describe());
+}
+
+@GuardedBy("lock")
+private void stopLeaderElectionCycle() {
+if (internalLeaderElector != null) {
+Preconditions.checkNotNull(currentLeaderElectionSession);
+
+// the current leader election cycle needs to be cancelled before
releasing the lock to
+// avoid retrying
+currentLeaderElectionSession.cancel(true);
+currentLeaderElectionSession = null;
+
+internalLeaderElector.release();
Review Comment:
Executing the `LeaderElector#release()` will trigger another `notLeader()`
callback, which revoke the leader contender leadership again. It might be
harmless because the leader contender already lost leadership.
Could you please share me why we do not simply add
`.withReleaseOnCancel(true)` when building the `leaderElectionConfig`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33264][table] Support source parallelism setting for DataGen connector [flink]
X-czh commented on code in PR #24133: URL: https://github.com/apache/flink/pull/24133#discussion_r1467174161 ## docs/content/docs/connectors/table/datagen.md: ## @@ -270,6 +270,13 @@ Connector Options Long The total number of rows to emit. By default, the table is unbounded. + + scan.parallelism + optional + (none) + Long Review Comment: Fixed in the new commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33264][table] Support source parallelism setting for DataGen connector [flink]
X-czh commented on code in PR #24133: URL: https://github.com/apache/flink/pull/24133#discussion_r1467171751 ## docs/content/docs/connectors/table/datagen.md: ## @@ -270,6 +270,13 @@ Connector Options Long The total number of rows to emit. By default, the table is unbounded. + + scan.parallelism + optional + (none) + Long Review Comment: Thanks for catching this. I'll fix that -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33264][table] Support source parallelism setting for DataGen connector [flink]
X-czh commented on code in PR #24133:
URL: https://github.com/apache/flink/pull/24133#discussion_r1467171607
##
flink-table/flink-table-api-java-bridge/src/test/java/org/apache/flink/table/factories/DataGenTableSourceFactoryTest.java:
##
@@ -581,6 +588,27 @@ void testWrongTypes() {
anyCauseMatches("Could not parse value 'Wrong' for key
'fields.f0.start'"));
}
+@Test
+void testWithParallelism() {
Review Comment:
I've already included an IT case in
`DataGeneratorConnectorITCase#testWithParallelism` where the physical execution
plan is checked, would that be enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33264][table] Support source parallelism setting for DataGen connector [flink]
libenchao commented on code in PR #24133:
URL: https://github.com/apache/flink/pull/24133#discussion_r1467163104
##
docs/content/docs/connectors/table/datagen.md:
##
@@ -270,6 +270,13 @@ Connector Options
Long
The total number of rows to emit. By default, the table is
unbounded.
+
+ scan.parallelism
+ optional
+ (none)
+ Long
Review Comment:
Actually it's defined as `int` in the code.
##
flink-table/flink-table-api-java-bridge/src/test/java/org/apache/flink/table/factories/DataGenTableSourceFactoryTest.java:
##
@@ -581,6 +588,27 @@ void testWrongTypes() {
anyCauseMatches("Could not parse value 'Wrong' for key
'fields.f0.start'"));
}
+@Test
+void testWithParallelism() {
Review Comment:
In FLINK-33263, we've introduced a way to verify that the transformation,
could you also add a test based on that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34196][Connectors][FLIP-389] Annotate SingleThreadFetcherManager as PublicEvolving. [flink]
loserwang1024 commented on PR #24171: URL: https://github.com/apache/flink/pull/24171#issuecomment-191130 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-33263) Implement ParallelismProvider for sources in the table planner
[ https://issues.apache.org/jira/browse/FLINK-33263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benchao Li resolved FLINK-33263. Fix Version/s: 1.19.0 Resolution: Fixed Implemented via 42b7e74ab20785289b62f5dd68d566995ba9dcfc (master) > Implement ParallelismProvider for sources in the table planner > -- > > Key: FLINK-33263 > URL: https://issues.apache.org/jira/browse/FLINK-33263 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Zhanghao Chen >Assignee: SuDewei >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33263][table-planner] Implement ParallelismProvider for sources in the table planner [flink]
libenchao closed pull request #24128: [FLINK-33263][table-planner] Implement ParallelismProvider for sources in the table planner URL: https://github.com/apache/flink/pull/24128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33365] include filters with Lookup joins [flink-connector-jdbc]
libenchao commented on PR #79: URL: https://github.com/apache/flink-connector-jdbc/pull/79#issuecomment-1911291438 > @libenchao a question to you before merging `squash` also removed the info about all the authors, are you ok if I add you as a co-author since it is based on your work? Or you can do it on your own and merge Sure, thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34206) CacheITCase.testRetryOnCorruptedClusterDataset(Path) failed
[
https://issues.apache.org/jira/browse/FLINK-34206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811084#comment-17811084
]
Zhu Zhu commented on FLINK-34206:
-
We will take a look.
> CacheITCase.testRetryOnCorruptedClusterDataset(Path) failed
> ---
>
> Key: FLINK-34206
> URL: https://issues.apache.org/jira/browse/FLINK-34206
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Blocker
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56728=logs=a657ddbf-d986-5381-9649-342d9c92e7fb=dc085d4a-05c8-580e-06ab-21f5624dab16=8763
> {code}
> Jan 23 01:39:48 01:39:48.152 [ERROR] Tests run: 6, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 19.24 s <<< FAILURE! -- in
> org.apache.flink.test.streaming.runtime.CacheITCase
> Jan 23 01:39:48 01:39:48.152 [ERROR]
> org.apache.flink.test.streaming.runtime.CacheITCase.testRetryOnCorruptedClusterDataset(Path)
> -- Time elapsed: 4.755 s <<< ERROR!
> Jan 23 01:39:48 org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> Jan 23 01:39:48 at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:144)
> Jan 23 01:39:48 at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$3(MiniClusterJobClient.java:141)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
> Jan 23 01:39:48 at
> org.apache.flink.runtime.rpc.pekko.PekkoInvocationHandler.lambda$invokeRpc$1(PekkoInvocationHandler.java:268)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
> Jan 23 01:39:48 at
> org.apache.flink.util.concurrent.FutureUtils.doForward(FutureUtils.java:1287)
> Jan 23 01:39:48 at
> org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$1(ClassLoadingUtils.java:93)
> Jan 23 01:39:48 at
> org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
> Jan 23 01:39:48 at
> org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$guardCompletionWithContextClassLoader$2(ClassLoadingUtils.java:92)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> Jan 23 01:39:48 at
> java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
> Jan 23 01:39:48 at
> org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$1.onComplete(ScalaFutureUtils.java:47)
> Jan 23 01:39:48 at
> org.apache.pekko.dispatch.OnComplete.internal(Future.scala:310)
> Jan 23 01:39:48 at
> org.apache.pekko.dispatch.OnComplete.internal(Future.scala:307)
> Jan 23 01:39:48 at
> org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:234)
> Jan 23 01:39:48 at
> org.apache.pekko.dispatch.japi$CallbackBridge.apply(Future.scala:231)
> Jan 23 01:39:48 at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
> Jan 23 01:39:48 at
> org.apache.flink.runtime.concurrent.pekko.ScalaFutureUtils$DirectExecutionContext.execute(ScalaFutureUtils.java:65)
> Jan 23 01:39:48 at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
> Jan 23 01:39:48 at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
> Jan 23 01:39:48 at
> scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
> Jan 23 01:39:48 at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
> Jan 23 01:39:48 at
> org.apache.pekko.pattern.PromiseActorRef.$bang(AskSupport.scala:629)
> Jan 23 01:39:48 at
[jira] [Updated] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-34237:
-
Fix Version/s: 1.19.0
(was: mongodb-1.1.0)
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Assignee: Wencong Liu
>Priority: Blocker
> Fix For: 1.19.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811082#comment-17811082
]
Xintong Song commented on FLINK-34237:
--
Thanks for reporting. This is indeed an unintended breaking change and a
blocker for Flink 1.19.
We thought `SourceReaderContext` is only called by various connectors and were
not aware of that it is also implemented by connectors. FLIP-382 is for
clean-up purposes and does not introduce any new feature. Even in Flink 2.0, I
think we should not require all connectors to change their codes only for such
clean-up purposes. So let's simply revert changes for this interface.
[~Wencong Liu], could you please help fix this?
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Priority: Blocker
> Fix For: mongodb-1.1.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song reassigned FLINK-34237:
Assignee: Wencong Liu
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Assignee: Wencong Liu
>Priority: Blocker
> Fix For: mongodb-1.1.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-34221) Introduce operator for minibatch join
[ https://issues.apache.org/jira/browse/FLINK-34221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dalongliu closed FLINK-34221. - Assignee: Shuai Xu Resolution: Fixed > Introduce operator for minibatch join > - > > Key: FLINK-34221 > URL: https://issues.apache.org/jira/browse/FLINK-34221 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Shuai Xu >Assignee: Shuai Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Introduce operator that implements minibatch join -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34221) Introduce operator for minibatch join
[ https://issues.apache.org/jira/browse/FLINK-34221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811081#comment-17811081 ] dalongliu commented on FLINK-34221: --- Merged in master: 8170c457cb70bb8fd88b98baf3acc612eaab8ec5 > Introduce operator for minibatch join > - > > Key: FLINK-34221 > URL: https://issues.apache.org/jira/browse/FLINK-34221 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Shuai Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Introduce operator that implements minibatch join -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34221) Introduce operator for minibatch join
[ https://issues.apache.org/jira/browse/FLINK-34221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-34221: --- Labels: pull-request-available (was: ) > Introduce operator for minibatch join > - > > Key: FLINK-34221 > URL: https://issues.apache.org/jira/browse/FLINK-34221 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Shuai Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Introduce operator that implements minibatch join -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-34221][table-runtime] Introduce MiniBatchStreamingJoinOperator [flink]
lsyldliu closed pull request #24160: [FLINK-34221][table-runtime] Introduce MiniBatchStreamingJoinOperator URL: https://github.com/apache/flink/pull/24160 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34100][table] Support session window table function without pulling up with window agg [flink]
xuyangzhong commented on code in PR #24162: URL: https://github.com/apache/flink/pull/24162#discussion_r1467143192 ## flink-table/flink-table-planner/src/test/resources/org/apache/flink/table/planner/plan/stream/sql/agg/WindowAggregateTest.xml: ## @@ -2170,6 +2360,386 @@ Sink(table=[default_catalog.default_database.sink], fields=[ws, we, b, c]) +- Exchange(distribution=[hash[b]]) +- Calc(select=[b, CASE((a = 11), 1, null:INTEGER) AS $f3, PROCTIME() AS proctime], where=[SEARCH(a, Sarg[1, 5, 7, 9, 11])]) +- TableSourceScan(table=[[default_catalog, default_database, source, project=[a, b], metadata=[]]], fields=[a, b]) +]]> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
[jira] [Updated] (FLINK-34238) In streaming mode, redundant exchange nodes can be optimally deleted in some cases
[
https://issues.apache.org/jira/browse/FLINK-34238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xuyang updated FLINK-34238:
---
Description:
Take the following plan as an example:
{code:java}
Calc(select=[window_start, window_end, a, EXPR$3, EXPR$4, EXPR$5, wAvg, uv])
+- WindowAggregate(groupBy=[a], window=[SESSION(win_start=[window_start],
win_end=[window_end], gap=[5 min], partition keys=[a])], select=[a, COUNT(*) AS
EXPR$3, SUM(d) AS EXPR$4, MAX(d) FILTER $f4 AS EXPR$5, weightedAvg(b, e) AS
wAvg, COUNT(DISTINCT c) AS uv, start('w$) AS window_start, end('w$) AS
window_end])
+- Exchange(distribution=[hash[a]])
+- Calc(select=[a, window_start, window_end, d, IS TRUE(>(b, 1000)) AS
$f4, b, e, c], where=[>=(window_start, 2021-01-01 10:10:00)])
+- WindowTableFunction(window=[SESSION(time_col=[rowtime], gap=[5
min], partition keys=[a])])
+- Exchange(distribution=[hash[a]])
+- WatermarkAssigner(rowtime=[rowtime], watermark=[-(rowtime,
1000:INTERVAL SECOND)])
+- TableSourceScan(table=[[default_catalog, default_database,
MyTable]], fields=[a, b, c, d, e, rowtime]) {code}
If the node `WindowTableFunction`, `Calc` and `WindowAggregate` can be chained
finally, the `Exchange` between `Calc` and `WindowAggregate` can be removed.
> In streaming mode, redundant exchange nodes can be optimally deleted in some
> cases
> --
>
> Key: FLINK-34238
> URL: https://issues.apache.org/jira/browse/FLINK-34238
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: xuyang
>Priority: Minor
>
> Take the following plan as an example:
> {code:java}
> Calc(select=[window_start, window_end, a, EXPR$3, EXPR$4, EXPR$5, wAvg, uv])
> +- WindowAggregate(groupBy=[a], window=[SESSION(win_start=[window_start],
> win_end=[window_end], gap=[5 min], partition keys=[a])], select=[a, COUNT(*)
> AS EXPR$3, SUM(d) AS EXPR$4, MAX(d) FILTER $f4 AS EXPR$5, weightedAvg(b, e)
> AS wAvg, COUNT(DISTINCT c) AS uv, start('w$) AS window_start, end('w$) AS
> window_end])
>+- Exchange(distribution=[hash[a]])
> +- Calc(select=[a, window_start, window_end, d, IS TRUE(>(b, 1000)) AS
> $f4, b, e, c], where=[>=(window_start, 2021-01-01 10:10:00)])
> +- WindowTableFunction(window=[SESSION(time_col=[rowtime], gap=[5
> min], partition keys=[a])])
> +- Exchange(distribution=[hash[a]])
>+- WatermarkAssigner(rowtime=[rowtime], watermark=[-(rowtime,
> 1000:INTERVAL SECOND)])
> +- TableSourceScan(table=[[default_catalog,
> default_database, MyTable]], fields=[a, b, c, d, e, rowtime]) {code}
> If the node `WindowTableFunction`, `Calc` and `WindowAggregate` can be
> chained finally, the `Exchange` between `Calc` and `WindowAggregate` can be
> removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34238) In streaming mode, redundant exchange nodes can be optimally deleted in some cases
xuyang created FLINK-34238: -- Summary: In streaming mode, redundant exchange nodes can be optimally deleted in some cases Key: FLINK-34238 URL: https://issues.apache.org/jira/browse/FLINK-34238 Project: Flink Issue Type: Improvement Components: Table SQL / Planner Reporter: xuyang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34233) HybridShuffleITCase.testHybridSelectiveExchangesRestart failed due to a IllegalStateException
[
https://issues.apache.org/jira/browse/FLINK-34233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811073#comment-17811073
]
Yunfeng Zhou commented on FLINK-34233:
--
Hi [~mapohl], truly this is also a bug caused by FLINK-33743. I'll look into
this.
> HybridShuffleITCase.testHybridSelectiveExchangesRestart failed due to a
> IllegalStateException
> -
>
> Key: FLINK-34233
> URL: https://issues.apache.org/jira/browse/FLINK-34233
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Priority: Blocker
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56791=logs=a596f69e-60d2-5a4b-7d39-dc69e4cdaed3=712ade8c-ca16-5b76-3acd-14df33bc1cb1=8357
> {code}
> Jan 24 02:10:03 02:10:03.582 [ERROR] Tests run: 12, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 34.74 s <<< FAILURE! -- in
> org.apache.flink.test.runtime.HybridShuffleITCase
> Jan 24 02:10:03 02:10:03.582 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridSelectiveExchangesRestart
> -- Time elapsed: 3.347 s <<< FAILURE!
> Jan 24 02:10:03 java.lang.AssertionError:
> org.apache.flink.runtime.JobException: org.apache.flink.runtime.JobException:
> Recovery is suppressed by
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2,
> backoffTimeMS=0)
> Jan 24 02:10:03 at
> org.apache.flink.test.runtime.JobGraphRunningUtil.execute(JobGraphRunningUtil.java:59)
> Jan 24 02:10:03 at
> org.apache.flink.test.runtime.BatchShuffleITCaseBase.executeJob(BatchShuffleITCaseBase.java:137)
> Jan 24 02:10:03 at
> org.apache.flink.test.runtime.HybridShuffleITCase.testHybridSelectiveExchangesRestart(HybridShuffleITCase.java:91)
> Jan 24 02:10:03 at
> java.base/java.lang.reflect.Method.invoke(Method.java:568)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
> Jan 24 02:10:03 at
> java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992)
> Jan 24 02:10:03 at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> Jan 24 02:10:03 at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Jan 24 02:10:03 at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
> Jan 24 02:10:03 at
> java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
> Jan 24 02:10:03 at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
> Jan 24 02:10:03 at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
> Jan 24 02:10:03 at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> Jan 24 02:10:03 at
> java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
> Jan 24 02:10:03 at
> java.base/java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:194)
> Jan 24 02:10:03 at
> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
> Jan 24 02:10:03 at
> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
> Jan 24 02:10:03 at
Re: [PR] [FLINK-34223][dist] Introduce a migration tool to transfer legacy config file to new config file. [flink]
JunRuiLee commented on PR #24185: URL: https://github.com/apache/flink/pull/24185#issuecomment-1911249227 Picked the latest fix https://github.com/apache/flink/pull/24194 and rebased onto master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34122][core] Deprecate old serialization config methods and options [flink]
X-czh commented on PR #24193: URL: https://github.com/apache/flink/pull/24193#issuecomment-1911227577 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33882] UT/IT for checkpointing statistics [flink]
JingGe commented on code in PR #24007:
URL: https://github.com/apache/flink/pull/24007#discussion_r1467074403
##
flink-runtime/src/test/java/org/apache/flink/runtime/metrics/DescriptiveStatisticsHistogramStatisticsTest.java:
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.metrics;
+
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests for {@link DescriptiveStatisticsHistogramStatistics} and {@link
+ * DescriptiveStatisticsHistogramStatistics.CommonMetricsSnapshot}.
+ */
+class DescriptiveStatisticsHistogramStatisticsTest {
+
+/** Tests the getPercentile functionality of the
DescriptiveStatisticsHistogramStatistics. */
Review Comment:
The intention of this test is to make sure the instance percentilesImpl does
not have null data.
##
flink-runtime/src/test/java/org/apache/flink/runtime/metrics/DescriptiveStatisticsHistogramStatisticsTest.java:
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.metrics;
+
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests for {@link DescriptiveStatisticsHistogramStatistics} and {@link
+ * DescriptiveStatisticsHistogramStatistics.CommonMetricsSnapshot}.
+ */
Review Comment:
The description contains no useful info. I would suggest remove it
completely.
##
flink-runtime/src/test/java/org/apache/flink/runtime/metrics/DescriptiveStatisticsHistogramStatisticsTest.java:
##
@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.metrics;
+
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/**
+ * Tests for {@link DescriptiveStatisticsHistogramStatistics} and {@link
+ * DescriptiveStatisticsHistogramStatistics.CommonMetricsSnapshot}.
+ */
+class DescriptiveStatisticsHistogramStatisticsTest {
+
+/** Tests the getPercentile functionality of the
DescriptiveStatisticsHistogramStatistics. */
+@Test
+void testDescriptiveStatisticsHistogramStatistics() {
+double data = 0.20;
+double percentileValue =
+new
DescriptiveStatisticsHistogramStatistics.CommonMetricsSnapshot()
+.getPercentile(data);
Review Comment:
maybe call CommonMetricsSnapshot.getValues() is better for the test purpose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
[jira] [Resolved] (FLINK-33843) Promote release 1.18.1
[
https://issues.apache.org/jira/browse/FLINK-33843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Ge resolved FLINK-33843.
-
Resolution: Fixed
> Promote release 1.18.1
> --
>
> Key: FLINK-33843
> URL: https://issues.apache.org/jira/browse/FLINK-33843
> Project: Flink
> Issue Type: New Feature
>Affects Versions: 1.18.0
>Reporter: Jing Ge
>Assignee: Jing Ge
>Priority: Major
> Labels: pull-request-available
>
> Once the release has been finalized, the last step of the process is to
> promote the release within the project and beyond. Please wait for 24h after
> finalizing the release in accordance with the [ASF release
> policy|http://www.apache.org/legal/release-policy.html#release-announcements].
> *Final checklist to declare this issue resolved:*
> # Website pull request to [list the
> release|http://flink.apache.org/downloads.html] merged
> # Release announced on the user@ mailing list.
> # Blog post published, if applicable.
> # Release recorded in
> [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink].
> # Release announced on social media.
> # Completion declared on the dev@ mailing list.
> # Update Homebrew:
> [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request] (seems to be done
> automatically - at least for minor releases for both minor and major
> releases)
> # Updated the japicmp configuration
> ** corresponding SNAPSHOT branch japicmp reference version set to the just
> released version, and API compatibiltity checks for {{@PublicEvolving}} was
> enabled
> ** (minor version release only) master branch japicmp reference version set
> to the just released version
> ** (minor version release only) master branch japicmp exclusions have been
> cleared
> # Update the list of previous version in {{docs/config.toml}} on the master
> branch.
> # Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch
> of the _now deprecated_ Flink version (i.e. 1.16 if 1.18.0 is released)
> # Update stable and master alias in
> [https://github.com/apache/flink/blob/master/.github/workflows/docs.yml]
> # Open discussion thread for End of Life for Unsupported version (i.e. 1.16)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-33838) Finalize release 1.18.1
[ https://issues.apache.org/jira/browse/FLINK-33838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Ge resolved FLINK-33838. - Resolution: Fixed > Finalize release 1.18.1 > --- > > Key: FLINK-33838 > URL: https://issues.apache.org/jira/browse/FLINK-33838 > Project: Flink > Issue Type: New Feature >Affects Versions: 1.18.0 >Reporter: Jing Ge >Assignee: Jing Ge >Priority: Major > > Once the release candidate has been reviewed and approved by the community, > the release should be finalized. This involves the final deployment of the > release candidate to the release repositories, merging of the website > changes, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-33842) Publish the Dockerfiles for the new release
[
https://issues.apache.org/jira/browse/FLINK-33842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Ge resolved FLINK-33842.
-
Resolution: Fixed
> Publish the Dockerfiles for the new release
> ---
>
> Key: FLINK-33842
> URL: https://issues.apache.org/jira/browse/FLINK-33842
> Project: Flink
> Issue Type: Sub-task
>Reporter: Jing Ge
>Assignee: Jing Ge
>Priority: Major
> Labels: pull-request-available
>
> Note: the official Dockerfiles fetch the binary distribution of the target
> Flink version from an Apache mirror. After publishing the binary release
> artifacts, mirrors can take some hours to start serving the new artifacts, so
> you may want to wait to do this step until you are ready to continue with the
> "Promote the release" steps in the follow-up Jira.
> Follow the [release instructions in the flink-docker
> repo|https://github.com/apache/flink-docker#release-workflow] to build the
> new Dockerfiles and send an updated manifest to Docker Hub so the new images
> are built and published.
> Note: the flink-web PR has to be merged before the pull request to be opened
> on the
> [{{official-images}}|https://github.com/docker-library/official-images] repo.
>
> h3. Expectations
> * Dockerfiles in [flink-docker|https://github.com/apache/flink-docker]
> updated for the new Flink release and pull request opened on the Docker
> official-images with an updated manifest
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33588) Fix Flink Checkpointing Statistics Bug
[ https://issues.apache.org/jira/browse/FLINK-33588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17798545#comment-17798545 ] Jing Ge edited comment on FLINK-33588 at 1/25/24 11:03 PM: --- [~zhutong66] Thanks for the feedback. Could you please elaborate on how you will create the test? The test should be part of the CI and used as regression tests. was (Author: jingge): [~zhutong66] Thanks for the feedback. Could you please elaborate on what you will create the test? The test should be part of the CI and used as regression tests. > Fix Flink Checkpointing Statistics Bug > -- > > Key: FLINK-33588 > URL: https://issues.apache.org/jira/browse/FLINK-33588 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.5, 1.16.0, 1.17.0, 1.15.2, 1.14.6, 1.18.0, 1.17.1 >Reporter: Tongtong Zhu >Assignee: Tongtong Zhu >Priority: Critical > Labels: pull-request-available > Fix For: 1.19.0, 1.18.1 > > Attachments: FLINK-33588.patch, image-2023-12-11-17-35-23-391.png, > image-2023-12-13-11-35-43-780.png, image-2023-12-15-13-59-28-201.png, > image-2023-12-19-14-02-49-083.png, image-2023-12-19-14-03-27-062.png, > newCommit-FLINK-33688.patch > > > When the Flink task is first started, the checkpoint data is null due to the > lack of data, and Percentile throws a null pointer exception when calculating > the percentage. After multiple tests, I found that it is necessary to set an > initial value for the statistical data value of the checkpoint when the > checkpoint data is null (i.e. at the beginning of the task) to solve this > problem. > The following is an abnormal description of the bug： > 2023-09-13 15:02:54,608 ERROR > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler > [] - Unhandled exception. > org.apache.commons.math3.exception.NullArgumentException: input array > at > org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1650) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:129) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:84) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleRequest(CheckpointingStatisticsHandler.java:58) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractAccessExecutionGraphHandler.handleRequest(AbstractAccessExecutionGraphHandler.java:68) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.lambda$handleRequest$0(AbstractExecutionGraphHandler.java:87) > ~[flink-dist_2.12-1.14.5.jar:1.14.5] > at > java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) > [?:1.8.0_151] > at > java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442) > [?:1.8.0_151] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_151] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151] > at >
[jira] [Commented] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811041#comment-17811041
]
Martijn Visser commented on FLINK-34237:
For context, we had a similar issue with FLINK-34149
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Priority: Blocker
> Fix For: mongodb-1.1.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-32743][Connectors/Kafka] Parse data from kafka connect and convert it into regular JSON data [flink-connector-kafka]
MartijnVisser commented on PR #42: URL: https://github.com/apache/flink-connector-kafka/pull/42#issuecomment-1911093338 > Thanks for your review, it has been processed CI is failing, will you take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33545) KafkaSink implementation can cause dataloss during broker issue when not using EXACTLY_ONCE if there's any batching
[
https://issues.apache.org/jira/browse/FLINK-33545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811037#comment-17811037
]
Martijn Visser commented on FLINK-33545:
[~mason6345] WDYT about [~aeolus811tw] his last comments?
> KafkaSink implementation can cause dataloss during broker issue when not
> using EXACTLY_ONCE if there's any batching
> ---
>
> Key: FLINK-33545
> URL: https://issues.apache.org/jira/browse/FLINK-33545
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.18.0
>Reporter: Kevin Tseng
>Assignee: Kevin Tseng
>Priority: Major
> Labels: pull-request-available
>
> In the current implementation of KafkaSource and KafkaSink there are some
> assumption that were made:
> # KafkaSource completely relies on Checkpoint to manage and track its offset
> in *KafkaSourceReader* class
> # KafkaSink in *KafkaWriter* class only performs catch-flush when
> *DeliveryGuarantee.EXACTLY_ONCE* is specified.
> KafkaSource is assuming that checkpoint should be properly fenced and
> everything it had read up-til checkpoint being initiated will be processed or
> recorded by operators downstream, including the TwoPhaseCommiter such as
> *KafkaSink*
> *KafkaSink* goes by the model of:
>
> {code:java}
> flush -> prepareCommit -> commit{code}
>
> In a scenario that:
> * KafkaSource ingested records #1 to #100
> * KafkaSink only had chance to send records #1 to #96
> * with a batching interval of 5ms
> when checkpoint has been initiated, flush will only confirm the sending of
> record #1 to #96.
> This allows checkpoint to proceed as there's no error, and record #97 to 100
> will be batched after first flush.
> Now, if broker goes down / has issue that caused the internal KafkaProducer
> to not be able to send out the record after a batch, and is on a constant
> retry-cycle (default value of KafkaProducer retries is Integer.MAX_VALUE),
> *WriterCallback* error handling will never be triggered until the next
> checkpoint flush.
> This can be tested by creating a faulty Kafka cluster and run the following
> code:
> {code:java}
> Properties props = new Properties();
> props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_SERVER);
> props.put(ProducerConfig.CLIENT_ID_CONFIG, "example-producer");
> props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
> StringSerializer.class.getName());
> props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
> props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, Integer.MAX_VALUE);
> props.put(ProducerConfig.ACKS_CONFIG, "all");
> final KafkaProducer producer = new KafkaProducer<>(props);
> try {
> for (int i = 0; i < 10; i++) {
> System.out.printf("sending record #%d\n", i);
> String data = UUID.randomUUID().toString();
> final ProducerRecord record = new
> ProducerRecord<>(TOPIC, Integer.toString(i), data);
> producer.send(record, new CB(Integer.toString(i), data));
> Thread.sleep(1); //sleep for 10 seconds
> }
> } catch (Exception e) {
> e.printStackTrace();
> } finally {
> System.out.println("flushing");
> producer.flush();
> System.out.println("closing");
> producer.close();
> }{code}
> Once callback returns due to network timeout, it will cause Flink to restart
> from previously saved checkpoint (which recorded reading up to record #100),
> but KafkaWriter never sent record #97 to #100.
> This will result in dataloss of record #97 to #100
> Because KafkaWriter only catches error *after* callback, if callback is never
> invoked (due to broker issue) right after the first flush has taken place,
> those records are effectively gone unless someone decided to go back and look
> for it.
> This behavior should be ok if user has set {*}DeliveryGuarantee.NONE{*}, but
> is not expected for {*}DeliveryGuarantee.AT_LEAST_ONCE{*}.
> There is a divergence of the process in the event of {*}EXACTLY_ONCE{*}.
> prepareCommit will produce a list of KafkaCommittable that corresponds to
> Transactional KafkaProducer to be committed. And a catch up flush will take
> place during *commit* step. Whether this was intentional or not, due to the
> fact that flush is a blocking call, the second flush for EXACTLY_ONCE at the
> end of EXACTLY_ONCE actually ensured everything fenced in the current
> checkpoint will be sent to Kafka, or fail the checkpoint if not successful.
>
> Due the above finding, I'm recommending one of the following fixes:
> # need to perform second flush for AT_LEAST_ONCE
> # or move flush to the end of the KafkaSink process.
> I'm
Re: [PR] [FLINK-33692][Connectors/Kafka] FlinkKafkaProducer could miss super.close [flink-connector-kafka]
MartijnVisser commented on PR #74: URL: https://github.com/apache/flink-connector-kafka/pull/74#issuecomment-1911089667 @mas-chen Any thoughts on this? Is this indeed a bug? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-25509][connector/kafka] Add RecordEvaluator to dynamically stop source based on de-serialized records [flink-connector-kafka]
MartijnVisser commented on PR #76: URL: https://github.com/apache/flink-connector-kafka/pull/76#issuecomment-1911088576 @mas-chen @ruanhang1993 I've brought this discussion (and my proposal) back to the Dev mailing list, see https://lists.apache.org/thread/0hp782dv6kjcv3ypthhfhyp7kglg83d0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811035#comment-17811035
]
Martijn Visser commented on FLINK-34237:
[~leonard] This looks like a blocker for Flink 1.19, not for MongoDB right?
Since the FLIP and the discussion thread didn't mention breaking anything in
Flink 1.19, only for Flink 2.0?
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Priority: Blocker
> Fix For: mongodb-1.1.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-32289) The metadata column type is incorrect in Kafka table connector example
[
https://issues.apache.org/jira/browse/FLINK-32289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-32289:
---
Fix Version/s: kafka-3.1.0
(was: kafka-4.0.0)
> The metadata column type is incorrect in Kafka table connector example
> --
>
> Key: FLINK-32289
> URL: https://issues.apache.org/jira/browse/FLINK-32289
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.15.4, 1.16.2, 1.17.1
>Reporter: Leonard Xu
>Assignee: Xiqian YU
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0, kafka-3.1.0
>
>
> The example[1] defined ts column with TIMESTAMP type
>
> {code:java}
> `ts` TIMESTAMP(3) METADATA FROM 'timestamp'
> {code}
> the correct column type should be TIMESTAMP_LTZ type.
>
> {code:java}
> `ts` TIMESTAMP_LTZ(3) METADATA FROM 'timestamp' {code}
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/table/kafka/#how-to-create-a-kafka-table
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
