[jira] [Commented] (FLINK-33170) HybridSourceSplitEnumerator causes dropped records in Hybrid Sources with 3+ sources
[
https://issues.apache.org/jira/browse/FLINK-33170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770650#comment-17770650
]
Robert Hoyt commented on FLINK-33170:
-
If someone can point me to how to Flink's policies on submitting a PR I can
take a shot at fixing this. Otherwise if someone else wants to tackle it, iirc
the fix was to clear the set [near the
end|https://github.com/apache/flink/blob/b25b57c55d903e4fdd2b666de49c90bfbad8fa99/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumerator.java#L305]
of the {{switchEnumerator}} method.
> HybridSourceSplitEnumerator causes dropped records in Hybrid Sources with 3+
> sources
>
>
> Key: FLINK-33170
> URL: https://issues.apache.org/jira/browse/FLINK-33170
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.15.4, 1.16.2, 1.17.1, 1.19.0, 1.18.1
>Reporter: Robert Hoyt
>Priority: Critical
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Possibly related to FLINK-27916.
>
> In all versions since 1.15.x there's a subtle bug in
> {{HybridSourceSplitEnumerator}} when determining if it's time to move on to
> the next source:
> {code:java}
> finishedReaders.add(subtaskId);
> if (finishedReaders.size() == context.currentParallelism()) {
> // move on to the next source if it exists
> {code}
> This snippet is correct, but when changing to the next source,
> {{finishedReaders}} is never cleared. So when processing the second source,
> the {{finishedReaders.size()}} check will return true when the _first_
> subtask finishes.** The hybrid source moves on to the next source if one
> exists, so any unsent records in other subtasks will get dropped.
>
> ** if each of the sources in the hybrid source has the same parallelism. If
> any source except the last has lower parallelism then I suspect that the
> source will never move on: it's impossible for `finishedReaders.size()` to
> shrink.
>
> Concrete example with three sources, two subtasks each:
> # subtask 0 finishes with the first source. {{finishedReaders}} has size 1
> # subtask 1 finishes with the first source. {{finishedReaders}} has size 2
> now, and moves on to the second source
> # subtask 1 finishes with the first source. {{finishedReaders.add(1)}}
> doesn't change the set; {{finishedReaders}} still has size 2. So the hybrid
> source moves on to the third source.
> # subtask 0 wasn't finished with the second source, but receives the
> notification to move on. Any unsent records are lost. *Data loss!*
> # this continues to the last source. The source doesn't change over if at
> the last source so the race condition in step 3 never happens
>
> So step 3 results in the race condition that will drop records
> indeterminately for all but the first source and last source.
> In production this issue caused the loss of GBs to TBs of data depending on
> the sources. We fixed it in a private fork by clearing the
> {{finishedReaders}} set when changing to the next source.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33170) HybridSourceSplitEnumerator causes dropped records in Hybrid Sources with 3+ sources
[
https://issues.apache.org/jira/browse/FLINK-33170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Hoyt updated FLINK-33170:
Description:
Possibly related to FLINK-27916.
In all versions since 1.15.x there's a subtle bug in
{{HybridSourceSplitEnumerator}} when determining if it's time to move on to the
next source:
{code:java}
finishedReaders.add(subtaskId);
if (finishedReaders.size() == context.currentParallelism()) {
// move on to the next source if it exists
{code}
This snippet is correct, but when changing to the next source,
{{finishedReaders}} is never cleared. So when processing the second source, the
{{finishedReaders.size()}} check will return true when the _first_ subtask
finishes.** The hybrid source moves on to the next source if one exists, so any
unsent records in other subtasks will get dropped.
** if each of the sources in the hybrid source has the same parallelism. If any
source except the last has lower parallelism then I suspect that the source
will never move on: it's impossible for `finishedReaders.size()` to shrink.
Concrete example with three sources, two subtasks each:
# subtask 0 finishes with the first source. {{finishedReaders}} has size 1
# subtask 1 finishes with the first source. {{finishedReaders}} has size 2
now, and moves on to the second source
# subtask 1 finishes with the first source. {{finishedReaders.add(1)}} doesn't
change the set; {{finishedReaders}} still has size 2. So the hybrid source
moves on to the third source.
# subtask 0 wasn't finished with the second source, but receives the
notification to move on. Any unsent records are lost. *Data loss!*
# this continues to the last source. The source doesn't change over if at the
last source so the race condition in step 3 never happens
So step 3 results in the race condition that will drop records indeterminately
for all but the first source and last source.
In production this issue caused the loss of GBs to TBs of data depending on the
sources. We fixed it in a private fork by clearing the {{finishedReaders}} set
when changing to the next source.
was:
Possibly related to FLINK-27916.
In all versions since 1.15.x there's a subtle bug in
HybridSourceSplitEnumerator when determining if it's time to move on to the
next source:
{code:java}
finishedReaders.add(subtaskId);
if (finishedReaders.size() == context.currentParallelism()) {
// move on to the next source if it exists
{code}
This snippet is correct, but when changing to the next source,
`finishedReaders` is never cleared. So when processing the second source, the
`finishedReaders.size()` check will return true when the _first_ subtask
finishes. The hybrid source moves on to the next source if one exists, so any
unsent records in other subtasks will get dropped.
Concrete example with three sources, two subtasks each:
# subtask 0 finishes with the first source. `finishedReaders` has size 1
# subtask 1 finishes with the first source. `finishedReaders` has size 2 now,
and moves on to the second source
# subtask 1 finishes with the first source. `finishedReaders.add(1)` doesn't
change the set; `finishedReaders` still has size 2. So the hybrid source moves
on to the third source.
# subtask 0 wasn't finished with the second source, but receives the
notification to move on. Any unsent records are lost. *Data loss.*
# this continues to the last source. The source doesn't change over if at the
last source so the race condition in step 3 never happens
So step 3 results in the race condition that will drop records indeterminately
for all but the first source and last source.
In production this issue caused the loss of GBs to TBs of data depending on the
sources. We fixed it in a private fork by clearing the `finishedReaders` set
when changing to the next source.
> HybridSourceSplitEnumerator causes dropped records in Hybrid Sources with 3+
> sources
>
>
> Key: FLINK-33170
> URL: https://issues.apache.org/jira/browse/FLINK-33170
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.15.4, 1.16.2, 1.17.1, 1.19.0, 1.18.1
>Reporter: Robert Hoyt
>Priority: Critical
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Possibly related to FLINK-27916.
>
> In all versions since 1.15.x there's a subtle bug in
> {{HybridSourceSplitEnumerator}} when determining if it's time to move on to
> the next source:
> {code:java}
> finishedReaders.add(subtaskId);
> if (finishedReaders.size() == context.currentParallelism()) {
> // move on to the next source if it exists
> {code}
> This snippet is correct, but when changing to the next source,
> {{finishedReaders}} is never cleared. So when processing the second source,
> the
[jira] [Comment Edited] (FLINK-33155) Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled
[
https://issues.apache.org/jira/browse/FLINK-33155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770648#comment-17770648
]
袁枫 edited comment on FLINK-33155 at 9/30/23 4:22 AM:
-
The root cause might be that we are reading the delegation token from JM local
file[1] ， as i know token will renew every 1 day by Yarn rm until reach 7days
limit, token in your log maxDate is ‘2023-09-27 15:53:51.487’， error ocurr is
'2023-9-25', it is say: althought jm use token in local file, it is also can be
sucesseful. If the token be renewed in 2023-9-25 sometime by yarn rm. So i
think if this is a yarn issue? [~wangyang0918], help your reply!
was (Author: feng yuan):
The root cause might be that we are reading the delegation token from JM local
file[1] ， as i know token will renew every 1 day by Yarn rm, token in your log
maxDate is ‘2023-09-27 15:53:51.487’， error ocurr is '2023-9-25', it is say:
althought jm use token in local file, it is also can be sucesseful. If the
token be renewed in 2023-9-25 sometime by yarn rm. So i think if this is a yarn
issue? [~wangyang0918], help your reply!
> Flink ResourceManager continuously fails to start TM container on YARN when
> Kerberos enabled
>
>
> Key: FLINK-33155
> URL: https://issues.apache.org/jira/browse/FLINK-33155
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Yang Wang
>Priority: Major
>
> When Kerberos enabled(with key tab) and after one day(the container token
> expired), Flink fails to create the TaskManager container on YARN due to the
> following exception.
>
> {code:java}
> 2023-09-25 16:48:50,030 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Worker container_1695106898104_0003_01_69 is terminated. Diagnostics:
> Container container_1695106898104_0003_01_69 was invalid. Diagnostics:
> [2023-09-25 16:48:45.710]token (token for hadoop: HDFS_DELEGATION_TOKEN
> owner=hadoop/master-1-1.c-5ee7bdc598b6e1cc.cn-beijing.emr.aliyuncs@emr.c-5ee7bdc598b6e1cc.com,
> renewer=, realUser=, issueDate=1695196431487, maxDate=1695801231487,
> sequenceNumber=12, masterKeyId=3) can't be found in cache
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (token for hadoop: HDFS_DELEGATION_TOKEN owner=, renewer=,
> realUser=, issueDate=1695196431487, maxDate=1695801231487, sequenceNumber=12,
> masterKeyId=3) can't be found in cache
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1545)
> at org.apache.hadoop.ipc.Client.call(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1666)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1573)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1588)
> at
> org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
> at
[jira] [Updated] (FLINK-33170) HybridSourceSplitEnumerator causes dropped records in Hybrid Sources with 3+ sources
[
https://issues.apache.org/jira/browse/FLINK-33170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Hoyt updated FLINK-33170:
Description:
Possibly related to FLINK-27916.
In all versions since 1.15.x there's a subtle bug in
HybridSourceSplitEnumerator when determining if it's time to move on to the
next source:
{code:java}
finishedReaders.add(subtaskId);
if (finishedReaders.size() == context.currentParallelism()) {
// move on to the next source if it exists
{code}
This snippet is correct, but when changing to the next source,
`finishedReaders` is never cleared. So when processing the second source, the
`finishedReaders.size()` check will return true when the _first_ subtask
finishes. The hybrid source moves on to the next source if one exists, so any
unsent records in other subtasks will get dropped.
Concrete example with three sources, two subtasks each:
# subtask 0 finishes with the first source. `finishedReaders` has size 1
# subtask 1 finishes with the first source. `finishedReaders` has size 2 now,
and moves on to the second source
# subtask 1 finishes with the first source. `finishedReaders.add(1)` doesn't
change the set; `finishedReaders` still has size 2. So the hybrid source moves
on to the third source.
# subtask 0 wasn't finished with the second source, but receives the
notification to move on. Any unsent records are lost. *Data loss.*
# this continues to the last source. The source doesn't change over if at the
last source so the race condition in step 3 never happens
So step 3 results in the race condition that will drop records indeterminately
for all but the first source and last source.
In production this issue caused the loss of GBs to TBs of data depending on the
sources. We fixed it in a private fork by clearing the `finishedReaders` set
when changing to the next source.
was:
Possibly related to FLINK-27916.
In all versions since 1.15.x there's a subtle bug
`HybridSourceSplitEnumerator`'s when determining if it's time to move on to the
next source:
```
finishedReaders.add(subtaskId);
if (finishedReaders.size() == context.currentParallelism()) {
// move on to the next source if it exists
```
This snippet is correct, but when changing to the next source,
`finishedReaders` is never cleared. So when processing the second source, the
`finishedReaders.size()` check will return true when the _first_ subtask
finishes. The hybrid source moves on to the next source if one exists, so any
unsent records in other subtasks will get dropped.
Concrete example with three sources, two subtasks each:
# subtask 0 finishes with the first source. `finishedReaders` has size 1
# subtask 1 finishes with the first source. `finishedReaders` has size 2 now,
and moves on to the second source
# subtask 1 finishes with the first source. `finishedReaders.add(1)` doesn't
change the set; `finishedReaders` still has size 2. So the hybrid source moves
on to the third source.
# subtask 0 wasn't finished with the second source, but receives the
notification to move on. Any unsent records are lost. *Data loss.*
# this continues to the last source. The source doesn't change over if at the
last source so the race condition in step 3 never happens
So step 3 results in the race condition that will drop records indeterminately
for all but the first source and last source.
In production this issue caused the loss of GBs to TBs of data depending on the
sources. We fixed it in a private fork by clearing the `finishedReaders` set
when changing to the next source.
> HybridSourceSplitEnumerator causes dropped records in Hybrid Sources with 3+
> sources
>
>
> Key: FLINK-33170
> URL: https://issues.apache.org/jira/browse/FLINK-33170
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Common
>Affects Versions: 1.15.4, 1.16.2, 1.17.1, 1.19.0, 1.18.1
>Reporter: Robert Hoyt
>Priority: Critical
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Possibly related to FLINK-27916.
>
> In all versions since 1.15.x there's a subtle bug in
> HybridSourceSplitEnumerator when determining if it's time to move on to the
> next source:
> {code:java}
> finishedReaders.add(subtaskId);
> if (finishedReaders.size() == context.currentParallelism()) {
> // move on to the next source if it exists
> {code}
> This snippet is correct, but when changing to the next source,
> `finishedReaders` is never cleared. So when processing the second source, the
> `finishedReaders.size()` check will return true when the _first_ subtask
> finishes. The hybrid source moves on to the next source if one exists, so any
> unsent records in other subtasks will get dropped.
>
> Concrete example with three sources, two subtasks
[jira] [Commented] (FLINK-33155) Flink ResourceManager continuously fails to start TM container on YARN when Kerberos enabled
[
https://issues.apache.org/jira/browse/FLINK-33155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770648#comment-17770648
]
袁枫 commented on FLINK-33155:
The root cause might be that we are reading the delegation token from JM local
file[1] ， as i know token will renew every 1 day by Yarn rm, token in your log
maxDate is ‘2023-09-27 15:53:51.487’， error ocurr is '2023-9-25', it is say:
althought jm use token in local file, it is also can be sucesseful. If the
token be renewed in 2023-9-25 sometime by yarn rm. So i think if this is a yarn
issue? [~wangyang0918], help your reply!
> Flink ResourceManager continuously fails to start TM container on YARN when
> Kerberos enabled
>
>
> Key: FLINK-33155
> URL: https://issues.apache.org/jira/browse/FLINK-33155
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Yang Wang
>Priority: Major
>
> When Kerberos enabled(with key tab) and after one day(the container token
> expired), Flink fails to create the TaskManager container on YARN due to the
> following exception.
>
> {code:java}
> 2023-09-25 16:48:50,030 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Worker container_1695106898104_0003_01_69 is terminated. Diagnostics:
> Container container_1695106898104_0003_01_69 was invalid. Diagnostics:
> [2023-09-25 16:48:45.710]token (token for hadoop: HDFS_DELEGATION_TOKEN
> owner=hadoop/master-1-1.c-5ee7bdc598b6e1cc.cn-beijing.emr.aliyuncs@emr.c-5ee7bdc598b6e1cc.com,
> renewer=, realUser=, issueDate=1695196431487, maxDate=1695801231487,
> sequenceNumber=12, masterKeyId=3) can't be found in cache
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (token for hadoop: HDFS_DELEGATION_TOKEN owner=, renewer=,
> realUser=, issueDate=1695196431487, maxDate=1695801231487, sequenceNumber=12,
> masterKeyId=3) can't be found in cache
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1545)
> at org.apache.hadoop.ipc.Client.call(Client.java:1491)
> at org.apache.hadoop.ipc.Client.call(Client.java:1388)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:907)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:431)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:166)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:158)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:96)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:362)
> at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1666)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1576)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1573)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1588)
> at
> org.apache.hadoop.yarn.util.FSDownload.verifyAndCopy(FSDownload.java:269)
> at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:67)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:414)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:411)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:411)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer$FSDownloadWrapper.doDownloadCall(ContainerLocalizer.java:243)
> at
>
[jira] [Commented] (FLINK-31689) Filesystem sink fails when parallelism of compactor operator changed
[
https://issues.apache.org/jira/browse/FLINK-31689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770637#comment-17770637
]
jirawech.s commented on FLINK-31689:
[~martijnvisser] Could you help with this PR, seems like no one review it
> Filesystem sink fails when parallelism of compactor operator changed
>
>
> Key: FLINK-31689
> URL: https://issues.apache.org/jira/browse/FLINK-31689
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.16.1
>Reporter: jirawech.s
>Assignee: jirawech.s
>Priority: Major
> Labels: pull-request-available
> Attachments: HelloFlinkHadoopSink.java
>
>
> I encounter this error when i tried to use Filesystem sink with Table SQL. I
> have not tested with Datastream API tho. You may refers to the error as below
> {code:java}
> // code placeholder
> java.util.NoSuchElementException
> at java.util.ArrayList$Itr.next(ArrayList.java:864)
> at
> org.apache.flink.connector.file.table.stream.compact.CompactOperator.initializeState(CompactOperator.java:119)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:286)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:700)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:676)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:643)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:917)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
> at java.lang.Thread.run(Thread.java:750) {code}
> I cannot attach the full reproducible code here, but you may follow my pseudo
> code in attachment and reproducible steps below
> 1. Create Kafka source
> 2. Set state.savepoints.dir
> 3. Set Job parallelism to 1
> 4. Create FileSystem Sink
> 5. Run the job and trigger savepoint with API
> {noformat}
> curl -X POST localhost:8081/jobs/:jobId/savepoints -d '{"cancel-job":
> false}'{noformat}
> {color:#172b4d}6. Cancel job, change parallelism to 2, and resume job from
> savepoint{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31689) Filesystem sink fails when parallelism of compactor operator changed
[
https://issues.apache.org/jira/browse/FLINK-31689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jirawech.s updated FLINK-31689:
---
Labels: pull-request-available (was: pull-request-available stale-assigned)
> Filesystem sink fails when parallelism of compactor operator changed
>
>
> Key: FLINK-31689
> URL: https://issues.apache.org/jira/browse/FLINK-31689
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.16.1
>Reporter: jirawech.s
>Assignee: jirawech.s
>Priority: Major
> Labels: pull-request-available
> Attachments: HelloFlinkHadoopSink.java
>
>
> I encounter this error when i tried to use Filesystem sink with Table SQL. I
> have not tested with Datastream API tho. You may refers to the error as below
> {code:java}
> // code placeholder
> java.util.NoSuchElementException
> at java.util.ArrayList$Itr.next(ArrayList.java:864)
> at
> org.apache.flink.connector.file.table.stream.compact.CompactOperator.initializeState(CompactOperator.java:119)
> at
> org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:122)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:286)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:106)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:700)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:676)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:643)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:948)
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:917)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:741)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:563)
> at java.lang.Thread.run(Thread.java:750) {code}
> I cannot attach the full reproducible code here, but you may follow my pseudo
> code in attachment and reproducible steps below
> 1. Create Kafka source
> 2. Set state.savepoints.dir
> 3. Set Job parallelism to 1
> 4. Create FileSystem Sink
> 5. Run the job and trigger savepoint with API
> {noformat}
> curl -X POST localhost:8081/jobs/:jobId/savepoints -d '{"cancel-job":
> false}'{noformat}
> {color:#172b4d}6. Cancel job, change parallelism to 2, and resume job from
> savepoint{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33170) HybridSourceSplitEnumerator causes dropped records in Hybrid Sources with 3+ sources
Robert Hoyt created FLINK-33170:
---
Summary: HybridSourceSplitEnumerator causes dropped records in
Hybrid Sources with 3+ sources
Key: FLINK-33170
URL: https://issues.apache.org/jira/browse/FLINK-33170
Project: Flink
Issue Type: Bug
Components: Connectors / Common
Affects Versions: 1.17.1, 1.16.2, 1.15.4, 1.19.0, 1.18.1
Reporter: Robert Hoyt
Possibly related to FLINK-27916.
In all versions since 1.15.x there's a subtle bug
`HybridSourceSplitEnumerator`'s when determining if it's time to move on to the
next source:
```
finishedReaders.add(subtaskId);
if (finishedReaders.size() == context.currentParallelism()) {
// move on to the next source if it exists
```
This snippet is correct, but when changing to the next source,
`finishedReaders` is never cleared. So when processing the second source, the
`finishedReaders.size()` check will return true when the _first_ subtask
finishes. The hybrid source moves on to the next source if one exists, so any
unsent records in other subtasks will get dropped.
Concrete example with three sources, two subtasks each:
# subtask 0 finishes with the first source. `finishedReaders` has size 1
# subtask 1 finishes with the first source. `finishedReaders` has size 2 now,
and moves on to the second source
# subtask 1 finishes with the first source. `finishedReaders.add(1)` doesn't
change the set; `finishedReaders` still has size 2. So the hybrid source moves
on to the third source.
# subtask 0 wasn't finished with the second source, but receives the
notification to move on. Any unsent records are lost. *Data loss.*
# this continues to the last source. The source doesn't change over if at the
last source so the race condition in step 3 never happens
So step 3 results in the race condition that will drop records indeterminately
for all but the first source and last source.
In production this issue caused the loss of GBs to TBs of data depending on the
sources. We fixed it in a private fork by clearing the `finishedReaders` set
when changing to the next source.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-33028) FLIP-348: Make expanding behavior of virtual metadata columns configurable
[ https://issues.apache.org/jira/browse/FLINK-33028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-33028. Resolution: Fixed > FLIP-348: Make expanding behavior of virtual metadata columns configurable > -- > > Key: FLINK-33028 > URL: https://issues.apache.org/jira/browse/FLINK-33028 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Fix For: 1.19.0 > > > Many SQL vendors expose additional metadata via so-called "pseudo columns" or > "system columns" next to the physical columns. > However, those columns should not be selected by default when expanding > SELECT *. Also for the sake of backward compatibility. Flink SQL already > offers pseudo columns next to the physical columns exposed as metadata > columns. > This proposal suggests to evolve the existing column design slightly to be > more useful for platform providers. > https://cwiki.apache.org/confluence/x/_o6zDw -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-33169) Window TVFs don't consider column expansion
[ https://issues.apache.org/jira/browse/FLINK-33169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-33169. Fix Version/s: 1.19.0 Resolution: Fixed Fixed in master: b25b57c55d903e4fdd2b666de49c90bfbad8fa99 > Window TVFs don't consider column expansion > --- > > Key: FLINK-33169 > URL: https://issues.apache.org/jira/browse/FLINK-33169 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Window TVFs don't consider the column expansion. The reason for this is that > `TABLE t` is expanded by a custom logic in the parser. The expansion logic > should consider the descriptor in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] alexey-lv commented on a diff in pull request #23477: [FLINK-33169] Consider descriptor information during system column expansion
alexey-lv commented on code in PR #23477:
URL: https://github.com/apache/flink/pull/23477#discussion_r1341861500
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/calcite/FlinkCalciteSqlValidator.java:
##
@@ -321,4 +332,131 @@ protected void addToSelectList(
// Always add to list
super.addToSelectList(list, aliases, fieldList, exp, scope,
includeSystemVars);
}
+
+@Override
+protected @PolyNull SqlNode performUnconditionalRewrites(
+@PolyNull SqlNode node, boolean underFrom) {
+
+// Special case for window TVFs like:
+// TUMBLE(TABLE t, DESCRIPTOR(metadata_virtual), INTERVAL '1' MINUTE))
+//
+// "TABLE t" is translated into an implicit "SELECT * FROM t". This
would ignore columns
+// that are not expanded by default. However, the descriptor
explicitly states the need
+// for this column. Therefore, explicit table expressions (for window
TVFs at most one)
+// are captured before rewriting and replaced with a "marker"
SqlSelect that contains the
+// descriptor information. The "marker" SqlSelect is considered during
column expansion.
+final List explicitTableArgs =
getExplicitTableOperands(node);
+
+final SqlNode rewritten = super.performUnconditionalRewrites(node,
underFrom);
+
+if (!(node instanceof SqlBasicCall)) {
+return rewritten;
+}
+final SqlBasicCall call = (SqlBasicCall) node;
+final SqlOperator operator = call.getOperator();
+
+if (operator instanceof SqlWindowTableFunction) {
+if (explicitTableArgs.stream().allMatch(Objects::isNull)) {
+return rewritten;
+}
+
+final List descriptors =
+call.getOperandList().stream()
+.filter(op -> op.getKind() == SqlKind.DESCRIPTOR)
+.flatMap(
+desc ->
+((SqlBasicCall) desc)
+.getOperandList().stream()
+
.filter(SqlIdentifier.class::isInstance)
+
.map(SqlIdentifier.class::cast))
+.collect(Collectors.toList());
+
+for (int i = 0; i < call.operandCount(); i++) {
+final SqlIdentifier tableArg = explicitTableArgs.get(i);
+if (tableArg != null) {
+call.setOperand(i, new ExplicitTableSqlSelect(tableArg,
descriptors));
+}
+}
+}
+
+return rewritten;
+}
+
+//
+// Column expansion
+//
+
+/**
+ * A special {@link SqlSelect} to capture the origin of a {@link
SqlKind#EXPLICIT_TABLE} within
+ * TVF operands.
+ */
+private static class ExplicitTableSqlSelect extends SqlSelect {
+
+private final List descriptors;
+
+public ExplicitTableSqlSelect(SqlIdentifier table, List
descriptors) {
+super(
+SqlParserPos.ZERO,
+null,
+SqlNodeList.of(SqlIdentifier.star(SqlParserPos.ZERO)),
+table,
+null,
+null,
+null,
+null,
+null,
+null,
+null,
+null);
+this.descriptors = descriptors;
+}
+}
+
+/**
+ * Returns whether the given column has been declared in a {@link
SqlKind#DESCRIPTOR} next to a
+ * {@link SqlKind#EXPLICIT_TABLE} within TVF operands.
+ */
+private static boolean declaredDescriptorColumn(SelectScope scope, Column
column) {
+if (!(scope.getNode() instanceof ExplicitTableSqlSelect)) {
+return false;
+}
+final ExplicitTableSqlSelect select = (ExplicitTableSqlSelect)
scope.getNode();
+return select.descriptors.stream()
+.map(SqlIdentifier::getSimple)
+.anyMatch(id -> id.equals(column.getName()));
+}
+
+/**
+ * Returns all {@link SqlKind#EXPLICIT_TABLE} operands within TVF
operands. A list entry is
+ * {@code null} if the operand is not an {@link SqlKind#EXPLICIT_TABLE}.
+ */
+private static List getExplicitTableOperands(SqlNode node) {
+if (!(node instanceof SqlBasicCall)) {
+return null;
+}
+final SqlBasicCall call = (SqlBasicCall) node;
+
+if (!(call.getOperator() instanceof SqlFunction)) {
+return null;
+}
+final SqlFunction function =
[jira] [Updated] (FLINK-33056) NettyClientServerSslTest#testValidSslConnection fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-33056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-33056:
---
Labels: auto-deprioritized-critical test-stability (was: stale-critical
test-stability)
Priority: Major (was: Critical)
This issue was labeled "stale-critical" 7 days ago and has not received any
updates so it is being deprioritized. If this ticket is actually Critical,
please raise the priority and ask a committer to assign you the issue or revive
the public discussion.
> NettyClientServerSslTest#testValidSslConnection fails on AZP
>
>
> Key: FLINK-33056
> URL: https://issues.apache.org/jira/browse/FLINK-33056
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Configuration, Runtime / Coordination
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
> Attachments: logs-cron_azure-test_cron_azure_core-1694048924.zip
>
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53020=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=8592
> fails with
> {noformat}
> Test testValidSslConnection[SSL provider =
> JDK](org.apache.flink.runtime.io.network.netty.NettyClientServerSslTest) is
> running.
>
> 01:20:31,479 [main] INFO
> org.apache.flink.runtime.io.network.netty.NettyConfig[] - NettyConfig
> [server address: localhost/127.0.0.1, server port range: 36717, ssl enabled:
> true, memory segment size (bytes): 1024, transport type: AUTO, number of
> server threads: 1 (manual), number of client thr
> eads: 1 (manual), server connect backlog: 0 (use Netty's default), client
> connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's
> default)]
> 01:20:31,479 [main] INFO
> org.apache.flink.runtime.io.network.netty.NettyServer[] - Transport
> type 'auto': using EPOLL.
> 01:20:31,475 [Flink Netty Client (42359) Thread 0] WARN
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline [] -
> An exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.
> org.apache.flink.shaded.netty4.io.netty.handler.codec.DecoderException:
> javax.net.ssl.SSLHandshakeException: server certificate with unknown
> fingerprint: CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown,
> C=Unknown
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
> ~[flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
> ~[flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
> org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
> [flink-shaded-netty-4.1.91.Final-17.0.jar:?]
> at
>
[jira] [Updated] (FLINK-31406) Do not delete jobgraph on scale only last-state upgrades
[ https://issues.apache.org/jira/browse/FLINK-31406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-31406: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Do not delete jobgraph on scale only last-state upgrades > > > Key: FLINK-31406 > URL: https://issues.apache.org/jira/browse/FLINK-31406 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available, stale-assigned > > Currently the operator always deletes the jobgraph from HA metadata so that > it's regenerated for last-state upgrades. > This is unnecessary for scale only operations. Keeping the jobgraph can > greately speed up startup time for some jobs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-16987) FLIP-95: Add new table source and sink interfaces
[ https://issues.apache.org/jira/browse/FLINK-16987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16987: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > FLIP-95: Add new table source and sink interfaces > - > > Key: FLINK-16987 > URL: https://issues.apache.org/jira/browse/FLINK-16987 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > Proper support for handling changelogs, more efficient processing of data > through the new Blink planner, and unified interfaces that are DataStream API > agnostic make it necessary to rework the table source and sink interfaces. > The goals of this FLIP are: > * *Simplify the current interface architecture*: > ** Merge upsert, retract, and append sinks. > ** Unify batch and streaming sources. > ** Unify batch and streaming sinks. > * *Allow sources to produce a changelog*: > ** UpsertTableSources have been requested a lot by users. Now is the time to > open the internal planner capabilities via the new interfaces. > ** According to FLIP-105, we would like to support changelogs for processing > formats such as [Debezium|https://debezium.io/]. > * *Don't rely on DataStream API for source and sinks*: > ** According to FLIP-32, the Table API and SQL should be independent of the > DataStream API which is why the `table-common` module has no dependencies on > `flink-streaming-java`. > ** Source and sink implementations should only depend on the `table-common` > module after FLIP-27. > ** Until FLIP-27 is ready, we still put most of the interfaces in > `table-common` and strictly separate interfaces that communicate with a > planner and actual runtime reader/writers. > * *Implement efficient sources and sinks without planner dependencies*: > ** Make Blink's internal data structures available to connectors. > ** Introduce stable interfaces for data structures that can be marked as > `@PublicEvolving`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #23477: [FLINK-33169] Consider descriptor information during system column expansion
flinkbot commented on PR #23477: URL: https://github.com/apache/flink/pull/23477#issuecomment-1741552430 ## CI report: * f87f3c33e5c345226efebdfbba51fe205f126a4f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33169) Window TVFs don't consider column expansion
[ https://issues.apache.org/jira/browse/FLINK-33169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33169: --- Labels: pull-request-available (was: ) > Window TVFs don't consider column expansion > --- > > Key: FLINK-33169 > URL: https://issues.apache.org/jira/browse/FLINK-33169 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > > Window TVFs don't consider the column expansion. The reason for this is that > `TABLE t` is expanded by a custom logic in the parser. The expansion logic > should consider the descriptor in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] twalthr opened a new pull request, #23477: [FLINK-33169] Consider descriptor information during system column expansion
twalthr opened a new pull request, #23477: URL: https://github.com/apache/flink/pull/23477 ## What is the purpose of the change "TABLE t" is translated into an implicit "SELECT * FROM t". This would ignore columns that are not expanded by default. However, the descriptor explicitly states the need for this column. ## Brief change log Explicit table expressions (for window TVFs at most one) are captured before rewriting and replaced with a "marker" SqlSelect that contains the descriptor information. The "marker" SqlSelect is considered during column expansion. ## Verifying this change This change added tests and can be verified as follows: `ColumnExpansionTest` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-29398) Utilize Rack Awareness in Flink Consumer
[ https://issues.apache.org/jira/browse/FLINK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-29398. --- Resolution: Fixed > Utilize Rack Awareness in Flink Consumer > > > Key: FLINK-29398 > URL: https://issues.apache.org/jira/browse/FLINK-29398 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Jeremy DeGroot >Assignee: Jeremy DeGroot >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: kafka-3.1.0 > > > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > was implemented some time ago in Kafka. This allows brokers and consumers to > communicate about the rack (or AWS Availability Zone) they're located in. > Reading from a local broker can save money in bandwidth and improve latency > for your consumers. > Flink Kafka consumers currently cannot easily use rack awareness if they're > deployed across multiple racks or availability zones, because they have no > control over which rack the Task Manager they'll be assigned to may be in. > This improvement proposes that a Kafka Consumer could be configured with a > callback or Future that could be run when it's being configured on the task > manager, that will set the appropriate value at runtime if a value is > provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29398) Utilize Rack Awareness in Flink Consumer
[ https://issues.apache.org/jira/browse/FLINK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai updated FLINK-29398: Fix Version/s: kafka-3.1.0 > Utilize Rack Awareness in Flink Consumer > > > Key: FLINK-29398 > URL: https://issues.apache.org/jira/browse/FLINK-29398 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Jeremy DeGroot >Assignee: Jeremy DeGroot >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: kafka-3.1.0 > > > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > was implemented some time ago in Kafka. This allows brokers and consumers to > communicate about the rack (or AWS Availability Zone) they're located in. > Reading from a local broker can save money in bandwidth and improve latency > for your consumers. > Flink Kafka consumers currently cannot easily use rack awareness if they're > deployed across multiple racks or availability zones, because they have no > control over which rack the Task Manager they'll be assigned to may be in. > This improvement proposes that a Kafka Consumer could be configured with a > callback or Future that could be run when it's being configured on the task > manager, that will set the appropriate value at runtime if a value is > provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29398) Utilize Rack Awareness in Flink Consumer
[ https://issues.apache.org/jira/browse/FLINK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770582#comment-17770582 ] Tzu-Li (Gordon) Tai commented on FLINK-29398: - Thanks for driving this to the finish line [~jeremy.degroot]. Merged to apache/flink-connector-kafka:main with d89a082180232bb79e3c764228c4e7dbb9eb6b8b > Utilize Rack Awareness in Flink Consumer > > > Key: FLINK-29398 > URL: https://issues.apache.org/jira/browse/FLINK-29398 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Jeremy DeGroot >Assignee: Jeremy DeGroot >Priority: Major > Labels: pull-request-available, stale-assigned > > [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] > was implemented some time ago in Kafka. This allows brokers and consumers to > communicate about the rack (or AWS Availability Zone) they're located in. > Reading from a local broker can save money in bandwidth and improve latency > for your consumers. > Flink Kafka consumers currently cannot easily use rack awareness if they're > deployed across multiple racks or availability zones, because they have no > control over which rack the Task Manager they'll be assigned to may be in. > This improvement proposes that a Kafka Consumer could be configured with a > callback or Future that could be run when it's being configured on the task > manager, that will set the appropriate value at runtime if a value is > provided. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-kafka] tzulitai closed pull request #53: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness
tzulitai closed pull request #53: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness URL: https://github.com/apache/flink-connector-kafka/pull/53 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] jeremy-degroot opened a new pull request, #53: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness
jeremy-degroot opened a new pull request, #53: URL: https://github.com/apache/flink-connector-kafka/pull/53 ## What is the purpose of the change This PR adds a new method to KafkaSourceBuilder that sets the `client.id` config for the KafkaSource to the value returned by the provided Supplier. It needs to be a Supplier because it needs to run on the TaskManager, and can't be determined at Job submit time like other configs. ## Brief change log - *Add setRackId to KafkaSourceBuilder* - *Plumb rackId into KafkaPartitionSplitReader* - *Add rack id tests* - *Document RackId feature* ## Verifying this change - *Added tests for the KafkaSplitReader that verify behaviors for null rackId Supplier, null and empty return values, and provided values.* - *Manually verified the change by running a 3-node cluster that covered two "racks" (AWS Availability Zones) against an Amazon MSK cluster.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? docs/Javadocs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] snuyanzin closed pull request #22412: [FLINK-31778][table-planner] Fix RowToRowCastRule
snuyanzin closed pull request #22412: [FLINK-31778][table-planner] Fix RowToRowCastRule URL: https://github.com/apache/flink/pull/22412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] snuyanzin commented on pull request #22412: [FLINK-31778][table-planner] Fix RowToRowCastRule
snuyanzin commented on PR #22412: URL: https://github.com/apache/flink/pull/22412#issuecomment-1741286141 Thanks for the contibution and sorry for the fact that the fix was merged under a different PR I didn't know about this one till today -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-31778) Casting array of rows produces incorrect result
[
https://issues.apache.org/jira/browse/FLINK-31778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin closed FLINK-31778.
---
Resolution: Duplicate
It looks like this issue has been fixed under
https://issues.apache.org/jira/browse/FLINK-32296
sorry I didn't know about this one till today
> Casting array of rows produces incorrect result
> ---
>
> Key: FLINK-31778
> URL: https://issues.apache.org/jira/browse/FLINK-31778
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.1, 1.15.4
>Reporter: Ilya Soin
>Assignee: Ilya Soin
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: image-2023-04-12-15-23-46-669.png, screenshot-2.png
>
>
> {code:java}
> select CAST(commissions AS ARRAY>) as commissions
>from (select ARRAY[ROW(123), ROW(234)] commissions){code}
> Expected output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(123.0), (234.0)] |
> +++
> {code}
> Actual output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(234.0), (234.0)] |
> +++ {code}
> Full working example:
> https://gist.github.com/soin08/5e0038dbefeba9192706e05a78ef3bc1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-connector-kafka] jeremy-degroot closed pull request #20: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness
jeremy-degroot closed pull request #20: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness URL: https://github.com/apache/flink-connector-kafka/pull/20 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] tzulitai commented on pull request #20: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness
tzulitai commented on PR #20: URL: https://github.com/apache/flink-connector-kafka/pull/20#issuecomment-1741283245 Actually, @jeremy-degroot would you be able to squash all your changes into one commit that is rebased on latest `main` branch? That would be super helpful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] tzulitai commented on a diff in pull request #20: [FLINK-29398][connector/kafka] Provide rack ID to Kafka Source to take advantage of Rack Awareness
tzulitai commented on code in PR #20:
URL:
https://github.com/apache/flink-connector-kafka/pull/20#discussion_r1341644229
##
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java:
##
@@ -355,6 +360,17 @@ public KafkaSourceBuilder setClientIdPrefix(String
prefix) {
return setProperty(KafkaSourceOptions.CLIENT_ID_PREFIX.key(), prefix);
}
+/**
+ * Set the clientRackId supplier to be passed down to the
KafkaPartitionSplitReader.
+ *
+ * @param rackIdCallback callback to provide Kafka consumer client.rack
+ * @return this KafkaSourceBuilder
+ */
+public KafkaSourceBuilder
setRackIdSupplier(SerializableSupplier rackIdCallback) {
+this.rackIdSupplier = rackIdCallback;
Review Comment:
nit: can we unify the naming here to just refer to this as the
`rackIdSupplier`?
##
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSource.java:
##
@@ -98,20 +100,24 @@ public class KafkaSource
private final KafkaRecordDeserializationSchema deserializationSchema;
// The configurations.
private final Properties props;
+// Client rackId callback
+private final SerializableSupplier rackIdSupplier;
KafkaSource(
KafkaSubscriber subscriber,
OffsetsInitializer startingOffsetsInitializer,
@Nullable OffsetsInitializer stoppingOffsetsInitializer,
Boundedness boundedness,
KafkaRecordDeserializationSchema deserializationSchema,
-Properties props) {
+Properties props,
+SerializableSupplier rackIdSupplier) {
Review Comment:
```suggestion
@Nullable SerializableSupplier rackIdSupplier) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-33169) Window TVFs don't consider column expansion
Timo Walther created FLINK-33169: Summary: Window TVFs don't consider column expansion Key: FLINK-33169 URL: https://issues.apache.org/jira/browse/FLINK-33169 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Timo Walther Assignee: Timo Walther Window TVFs don't consider the column expansion. The reason for this is that `TABLE t` is expanded by a custom logic in the parser. The expansion logic should consider the descriptor in this case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint
[ https://issues.apache.org/jira/browse/FLINK-30238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770492#comment-17770492 ] Tzu-Li (Gordon) Tai commented on FLINK-30238: - [~martijnvisser] I've replied to your SinkFunction deprecation / SinkV2 public thread in the ML, and touched on this issue as an attempt to revive the discussion there. Lets move the discussion there and let me know what you think. > Unified Sink committer does not clean up state on final savepoint > - > > Key: FLINK-30238 > URL: https://issues.apache.org/jira/browse/FLINK-30238 > Project: Flink > Issue Type: Bug > Components: Connectors / Common >Affects Versions: 1.17.0, 1.15.3, 1.16.1 >Reporter: Fabian Paul >Priority: Critical > Attachments: Screenshot 2023-03-09 at 1.47.11 PM.png, image (8).png > > > During stop-with-savepoint the committer only commits the pending > committables on notifyCheckpointComplete. > This has several downsides. > * Last committableSummary has checkpoint id LONG.MAX and is never cleared > from the state leading to that stop-with-savepoint does not work when the > pipeline recovers from a savepoint > * While the committables are committed during stop-with-savepoint they are > not forwarded to post-commit topology, potentially losing data and preventing > to close open transactions. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-33028) FLIP-348: Make expanding behavior of virtual metadata columns configurable
[ https://issues.apache.org/jira/browse/FLINK-33028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther reopened FLINK-33028: -- > FLIP-348: Make expanding behavior of virtual metadata columns configurable > -- > > Key: FLINK-33028 > URL: https://issues.apache.org/jira/browse/FLINK-33028 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Fix For: 1.19.0 > > > Many SQL vendors expose additional metadata via so-called "pseudo columns" or > "system columns" next to the physical columns. > However, those columns should not be selected by default when expanding > SELECT *. Also for the sake of backward compatibility. Flink SQL already > offers pseudo columns next to the physical columns exposed as metadata > columns. > This proposal suggests to evolve the existing column design slightly to be > more useful for platform providers. > https://cwiki.apache.org/confluence/x/_o6zDw -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33089) Drop Flink 1.13 support
[ https://issues.apache.org/jira/browse/FLINK-33089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-33089: --- Summary: Drop Flink 1.13 support (was: Drop Flink 1.14 support) > Drop Flink 1.13 support > --- > > Key: FLINK-33089 > URL: https://issues.apache.org/jira/browse/FLINK-33089 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Gyula Fora >Priority: Major > Fix For: kubernetes-operator-1.7.0 > > > As agreed with the community we will only support the last 4 stable Flink > minor versions. > With Flink 1.17 already out, we should drop 1.13 support from the operator. > This includes any special codepaths required and we should probably throw a > validation error and short-circuit reconciliation on unsupported versions to > signal to users and avoid any accidental deployment problems. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33156) Remove flakiness from tests in OperatorStateBackendTest.java
[
https://issues.apache.org/jira/browse/FLINK-33156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33156:
---
Labels: pull-request-available (was: )
> Remove flakiness from tests in OperatorStateBackendTest.java
>
>
> Key: FLINK-33156
> URL: https://issues.apache.org/jira/browse/FLINK-33156
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.17.1
>Reporter: Asha Boyapati
>Assignee: Asha Boyapati
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.17.1
>
>
> This issue is similar to:
> https://issues.apache.org/jira/browse/FLINK-32963
> We are proposing to make the following tests stable:
> {quote}org.apache.flink.runtime.state.OperatorStateBackendTest#testSnapshotRestoreSync
> org.apache.flink.runtime.state.OperatorStateBackendTest#testSnapshotRestoreAsync{quote}
> The tests are currently flaky because the order of elements returned by
> iterators is non-deterministic.
> The following PR fixes the flaky test by making them independent of the order
> of elements returned by the iterator:
> https://github.com/apache/flink/pull/23464
> We detected this using the NonDex tool using the following commands:
> {quote}mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -pl flink-runtime
> -DnondexRuns=10
> -Dtest=org.apache.flink.runtime.state.OperatorStateBackendTest#testSnapshotRestoreSync
> mvn edu.illinois:nondex-maven-plugin:2.1.1:nondex -pl flink-runtime
> -DnondexRuns=10
> -Dtest=org.apache.flink.runtime.state.OperatorStateBackendTest#testSnapshotRestoreAsync{quote}
> Please see the following Continuous Integration log that shows the flakiness:
> https://github.com/asha-boyapati/flink/actions/runs/6193757385
> Please see the following Continuous Integration log that shows that the
> flakiness is fixed by this change:
> https://github.com/asha-boyapati/flink/actions/runs/619409
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] asha-boyapati commented on pull request #23464: [FLINK-33156][Test]Remove flakiness from tests in OperatorStateBackendTest.java
asha-boyapati commented on PR #23464: URL: https://github.com/apache/flink/pull/23464#issuecomment-1741011694 Thanks for your comments. I updated the title. I will look into your suggestion of finding and fixing similar problems together. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31778) Casting array of rows produces incorrect result
[
https://issues.apache.org/jira/browse/FLINK-31778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770463#comment-17770463
]
david radley commented on FLINK-31778:
--
[~soin08] It looks like this is fixed in master
[https://github.com/apache/flink/commit/6d62f9918ea2cbb8a10c705a25a4ff6deab60711]
- we should close out this issue.
> Casting array of rows produces incorrect result
> ---
>
> Key: FLINK-31778
> URL: https://issues.apache.org/jira/browse/FLINK-31778
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.1, 1.15.4
>Reporter: Ilya Soin
>Assignee: Ilya Soin
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: image-2023-04-12-15-23-46-669.png, screenshot-2.png
>
>
> {code:java}
> select CAST(commissions AS ARRAY>) as commissions
>from (select ARRAY[ROW(123), ROW(234)] commissions){code}
> Expected output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(123.0), (234.0)] |
> +++
> {code}
> Actual output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(234.0), (234.0)] |
> +++ {code}
> Full working example:
> https://gist.github.com/soin08/5e0038dbefeba9192706e05a78ef3bc1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] gaborgsomogyi commented on pull request #23417: [FLINK-33030][python]Add python 3.11 support
gaborgsomogyi commented on PR #23417: URL: https://github.com/apache/flink/pull/23417#issuecomment-1740951700 Rebased to the latest master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] davidradl commented on pull request #22412: [FLINK-31778][table-planner] Fix RowToRowCastRule
davidradl commented on PR #22412: URL: https://github.com/apache/flink/pull/22412#issuecomment-1740879352 @soin08 hi , we are interested in getting this fix in - are you in a position to fix the conflicts that are now present. Manyt thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31778) Casting array of rows produces incorrect result
[
https://issues.apache.org/jira/browse/FLINK-31778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770414#comment-17770414
]
david radley commented on FLINK-31778:
--
[~nictownsend] I have done a test on the scenario you describe in your comment
with the fix on and it does fix this scenario as well.
> Casting array of rows produces incorrect result
> ---
>
> Key: FLINK-31778
> URL: https://issues.apache.org/jira/browse/FLINK-31778
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.1, 1.15.4
>Reporter: Ilya Soin
>Assignee: Ilya Soin
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: image-2023-04-12-15-23-46-669.png, screenshot-2.png
>
>
> {code:java}
> select CAST(commissions AS ARRAY>) as commissions
>from (select ARRAY[ROW(123), ROW(234)] commissions){code}
> Expected output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(123.0), (234.0)] |
> +++
> {code}
> Actual output:
> {code:java}
> +++
> | op | commissions |
> +++
> | +I | [(234.0), (234.0)] |
> +++ {code}
> Full working example:
> https://gist.github.com/soin08/5e0038dbefeba9192706e05a78ef3bc1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-web] rmetzger commented on pull request #676: [FLINK-33046][FLIP-333] Redesign Apache Flink website
rmetzger commented on PR #676: URL: https://github.com/apache/flink-web/pull/676#issuecomment-1740784269 Why is "Powered By" in the footer and not in the "About" menu? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RainerMatthiasS commented on pull request #23239: [FLINK-26585][state-processor-api] replace implementation of MultiStateKeyIterator with Stream-free implementation
RainerMatthiasS commented on PR #23239: URL: https://github.com/apache/flink/pull/23239#issuecomment-1740768001 Hi @masteryhx , thank you for the review again, I also synced my fork with the upstream, if that was what you meant by 'Could you also rebase all your commits to the master ?' BTW: removing `boolean hasnext = ` created an additional warning on build, that's why I added it in the first place. Sincere greeting Thias -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a diff in pull request #676: [FLINK-33046][FLIP-333] Redesign Apache Flink website
rmetzger commented on code in PR #676: URL: https://github.com/apache/flink-web/pull/676#discussion_r1341269756 ## docs/i18n/zh.yaml: ## @@ -0,0 +1,163 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at + +# http://www.apache.org/licenses/LICENSE-2.0 + +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +- id: hero.title + translation: Apache Flink + +- id: hero.subtitle + translation: 数据流上的有状态计算 + +- id: hero.blurb + translation: Apache Flink 是一个框架和分布式处理引擎，用于在无边界和有边界数据流上进行有状态的计算。Flink 能在所有常见集群环境中运行，并能以内存速度和任意规模进行计算。 + +- id: learn-more + translation: 了解更多 + +- id: flink-capabilities + translation: Flink Capabilities + +- id: correctness-guarentees + translation: 正确性保证 + +- id: correctness-guarentees.exactly-once + translation: Exactly-once 状态一致性 + +- id: correctness-guarentees.event-time + translation: 事件时间处理 + +- id: correctness-guarentees.late-data-handling + translation: 成熟的迟到数据处理 + +- id: layered-apis + translation: 分层 API + +- id: layered-apis.sql + translation: SQL on Stream & Batch Data + +- id: layered-apis.data-stream-set + translation: DataStream API & DataSet API + +- id: layered-apis.process-function + translation: ProcessFunction (Time & State) + +- id: operational-focus + translation: 聚焦运维 + +- id: operational-focus.flexible-deployment + translation: 灵活部署 + +- id: operational-focus.high-availability + translation: 高可用 + +- id: operational-focus.savepoints + translation: 保存点 + +- id: scalability + translation: 大规模计算 + +- id: scalability.scale-out + translation: 水平扩展架构 + +- id: scalability.large-state + translation: 支持超大状态 + +- id: scalability.incremental-checkpoints + translation: 增量检查点机制 + +- id: performance + translation: 性能卓越 + +- id: performance.low-latency + translation: 低延迟 + +- id: performance.high-throughput + translation: 高吞吐 + +- id: performance.in-memory + translation: 内存计算 + +- id: use-cases + translation: 所有流式场景 + +- id: use-cases.event-driven + translation: 事件驱动应用 + +- id: use-cases.event-driven.description + translation: 事件驱动型应用是一类具有状态的应用，它从一个或多个事件流提取数据，并根据到来的事件触发计算、状态更新或其他外部动作。 + +- id: use-cases.stream-batch + translation: 流批分析 + +- id: use-cases.stream-batch.description + translation: 数据分析任务需要从原始数据中提取有价值的信息和指标。传统的分析方式通常是利用批查询，或将事件记录下来并基于此有限数据集构建应用来完成。 + +- id: use-cases.pipelines-etl + translation: 数据管道 & ETL + +- id: use-cases.pipelines-etl.description + translation: 提取-转换-加载（ETL）是一种在存储系统之间进行数据转换和迁移的常用方法。 + +- id: use-cases.learn-more + translation: 了解Flink的应用场景 + +- id: blog-posts + translation: 最新博客列表 + +- id: footer.disclaimer + translation: The contents of this website are © 2023 Apache Software Foundation under the terms of the Apache License v2. Apache Flink, Flink, and the Flink logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. + +- id: footer.powered-by + translation: Flink 用户 + +- id: footer.flink-package + translation: flink-package.org Review Comment: ```suggestion translation: flink-packages.org ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a diff in pull request #676: [FLINK-33046][FLIP-333] Redesign Apache Flink website
rmetzger commented on code in PR #676: URL: https://github.com/apache/flink-web/pull/676#discussion_r1341269428 ## docs/i18n/en.yaml: ## @@ -0,0 +1,163 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at + +# http://www.apache.org/licenses/LICENSE-2.0 + +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +- id: hero.title + translation: Apache Flink + +- id: hero.subtitle + translation: Stateful Computations over Data Streams + +- id: hero.blurb + translation: Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. + +- id: learn-more + translation: Learn More + +- id: flink-capabilities + translation: Flink Capabilities + +- id: correctness-guarentees + translation: Correctness guarantees + +- id: correctness-guarentees.exactly-once + translation: Exactly-once state consistency + +- id: correctness-guarentees.event-time + translation: Event-time processing + +- id: correctness-guarentees.late-data-handling + translation: Sophisticated late data handling + +- id: layered-apis + translation: Layered APIs + +- id: layered-apis.sql + translation: SQL on Stream & Batch Data + +- id: layered-apis.data-stream-set + translation: DataStream API & DataSet API + +- id: layered-apis.process-function + translation: ProcessFunction (Time & State) + +- id: operational-focus + translation: Operational focus + +- id: operational-focus.flexible-deployment + translation: Flexible deployment + +- id: operational-focus.high-availability + translation: High-availability setup + +- id: operational-focus.savepoints + translation: Savepoints + +- id: scalability + translation: Scalability + +- id: scalability.scale-out + translation: Scale-out architecture + +- id: scalability.large-state + translation: Support for very large state + +- id: scalability.incremental-checkpoints + translation: Incremental Checkpoints + +- id: performance + translation: Performance + +- id: performance.low-latency + translation: Low latency + +- id: performance.high-throughput + translation: High throughput + +- id: performance.in-memory + translation: In-Memory computing + +- id: use-cases + translation: Use Cases + +- id: use-cases.event-driven + translation: Event Driven Applications + +- id: use-cases.event-driven.description + translation: An event-driven application is a stateful application that ingest events from one or more event streams and reacts to incoming events by triggering computations, state updates, or external actions. + +- id: use-cases.stream-batch + translation: Stream & Batch Analytics + +- id: use-cases.stream-batch.description + translation: Analytical jobs extract information and insight from raw data. Traditionally, analytics are performed as batch queries or applications on bounded data sets of recorded events. + +- id: use-cases.pipelines-etl + translation: Data Pipelines & ETL + +- id: use-cases.pipelines-etl.description + translation: Extract-transform-load (ETL) is a common approach to convert and move data between storage systems. + +- id: use-cases.learn-more + translation: Learn more about Flink use cases + +- id: blog-posts + translation: Recent Flink blogs + +- id: footer.disclaimer + translation: The contents of this website are © 2023 Apache Software Foundation under the terms of the Apache License v2. Apache Flink, Flink, and the Flink logo are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. + +- id: footer.powered-by + translation: Powered by + +- id: footer.flink-package + translation: flink-package.org Review Comment: ```suggestion translation: flink-packages.org ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770398#comment-17770398 ] Gabor Somogyi commented on FLINK-32223: --- The normal build passed against master: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53521=results Waiting on nightly... > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-web] rmetzger commented on a diff in pull request #676: [FLINK-33046][FLIP-333] Redesign Apache Flink website
rmetzger commented on code in PR #676: URL: https://github.com/apache/flink-web/pull/676#discussion_r1341267362 ## docs/i18n/en.yaml: ## @@ -0,0 +1,163 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at + +# http://www.apache.org/licenses/LICENSE-2.0 + +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +- id: hero.title + translation: Apache Flink Review Comment: The old website is using the trademark (r) for the first mention of Apache Flink on the website. In my understanding of the trademark usage rules, we should do that on our own website. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] dannycranmer merged pull request #677: Update powered-by page to rename AWS service
dannycranmer merged PR #677: URL: https://github.com/apache/flink-web/pull/677 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #23443: [FLINK-33059] Support transparent compression for file-connector for all file input formats
echauchot commented on PR #23443: URL: https://github.com/apache/flink/pull/23443#issuecomment-1740565364 R: @xintongsong I see your name in that code history, would you have time to take a look ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33104) Nightly run for Flink Kafka connector fails due to architecture tests failing
[
https://issues.apache.org/jira/browse/FLINK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17770338#comment-17770338
]
Etienne Chauchot commented on FLINK-33104:
--
[~martijnvisser] actually it was never fixed in cassandra (see FLINK-32353):
the problem is that when archunit rules change we need to update the archunit
violation store. And there is only a single violation store. As the nightly
tests the connector against several versions of flink, there will be failures
if not all these versions have the same archunit rules. To fix this problem we
need to skip archunit tests on Flink versions that are not the main one (the
one the connector is built against) so that the single arunit violation store
contains the violations for the main flink version. I proposed some change in
the github action script here for which I was waiting for your feedback.
> Nightly run for Flink Kafka connector fails due to architecture tests failing
> -
>
> Key: FLINK-33104
> URL: https://issues.apache.org/jira/browse/FLINK-33104
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: kafka-3.1.0
>Reporter: Martijn Visser
>Priority: Blocker
>
> {code:java}
> 2023-09-17T00:29:07.1675694Z [WARNING] Tests run: 18, Failures: 0, Errors: 0,
> Skipped: 9, Time elapsed: 308.532 s - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerMigrationTest
> 2023-09-17T00:29:07.5171608Z [INFO]
> 2023-09-17T00:29:07.5172360Z [INFO] Results:
> 2023-09-17T00:29:07.5172773Z [INFO]
> 2023-09-17T00:29:07.5173139Z [ERROR] Failures:
> 2023-09-17T00:29:07.5174181Z [ERROR] Architecture Violation [Priority:
> MEDIUM] - Rule 'ITCASE tests should use a MiniCluster resource or extension'
> was violated (13 times):
> 2023-09-17T00:29:07.5176050Z
> org.apache.flink.connector.kafka.sink.FlinkKafkaInternalProducerITCase does
> not satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5177452Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5179831Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5181277Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5182154Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5182951Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5183906Z
> org.apache.flink.connector.kafka.sink.KafkaSinkITCase does not satisfy: only
> one of the following predicates match:
> 2023-09-17T00:29:07.5184769Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5185812Z * reside outside of package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type MiniClusterExtension and annotated with @RegisterExtension or are
> , and of type MiniClusterTestEnvironment and annotated with @TestEnv
> 2023-09-17T00:29:07.5186880Z * reside in a package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> InternalMiniClusterExtension
> 2023-09-17T00:29:07.5187929Z * reside outside of package
> 'org.apache.flink.runtime.*' and is annotated with @ExtendWith with class
> MiniClusterExtension
> 2023-09-17T00:29:07.5189073Z or contain any fields that are public, static,
> and of type MiniClusterWithClientResource and final and annotated with
> @ClassRule or contain any fields that is of type
> MiniClusterWithClientResource and public and final and not static and
> annotated with @Rule
> 2023-09-17T00:29:07.5190076Z
> org.apache.flink.connector.kafka.sink.KafkaTransactionLogITCase does not
> satisfy: only one of the following predicates match:
> 2023-09-17T00:29:07.5190946Z * reside in a package
> 'org.apache.flink.runtime.*' and contain any fields that are static, final,
> and of type InternalMiniClusterExtension and annotated with @RegisterExtension
> 2023-09-17T00:29:07.5191983Z * reside
[jira] [Closed] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi closed FLINK-32223. - > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (FLINK-32223) Add Hive delegation token support
[ https://issues.apache.org/jira/browse/FLINK-32223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved FLINK-32223. --- Fix Version/s: 1.19.0 Resolution: Fixed e4c15aa on master > Add Hive delegation token support > -- > > Key: FLINK-32223 > URL: https://issues.apache.org/jira/browse/FLINK-32223 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hadoop Compatibility >Reporter: qingbo jiao >Assignee: qingbo jiao >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.19.0 > > Attachments: image-2023-08-03-10-18-56-029.png, > image-2023-08-03-10-24-53-860.png, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] gaborgsomogyi commented on pull request #23127: [FLINK-32223][runtime][security] Add Hive delegation token support
gaborgsomogyi commented on PR #23127: URL: https://github.com/apache/flink/pull/23127#issuecomment-1740510003 We must track the nightly + all other builds not to break something. I'm going to track things but I'm ask you @jiaoqingbo to do it also. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaborgsomogyi merged pull request #23127: [FLINK-32223][runtime][security] Add Hive delegation token support
gaborgsomogyi merged PR #23127: URL: https://github.com/apache/flink/pull/23127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a diff in pull request #23223: [FLINK-32873][table] - Add config to allow disabling QUERY hints
dawidwys commented on code in PR #23223: URL: https://github.com/apache/flink/pull/23223#discussion_r1340991937 ## docs/layouts/shortcodes/generated/table_config_configuration.html: ## @@ -62,6 +62,12 @@ String The local time zone defines current session time zone id. It is used when converting to/from codeTIMESTAMP WITH LOCAL TIME ZONE/code. Internally, timestamps with local time zone are always represented in the UTC time zone. However, when converting to data types that don't include a time zone (e.g. TIMESTAMP, TIME, or simply STRING), the session time zone is used during conversion. The input of option is either a full name such as "America/Los_Angeles", or a custom timezone id such as "GMT-08:00". + +table.optimizer.query-hints.enabled Batch Streaming Review Comment: cc @wuchong Hey Jark, I am reaching out to you since you suggested the `table.optimizer.ignore-query-hints`. I think Bonnie has a good point here that `table.optimizer.query-hints.enabled` would be more consistent with other configs: ``` table.dynamic-table-options.enabled table.optimizer.distinct-agg.split.enabled table.optimizer.dynamic-filtering.enabled ... ``` Moreover, I think `ignore-query-hints` would be a bit misleading. From my point of view it would suggest query hints are simply ignored, but from what I've gathered the agreed behaviour is to throw an exception which is not necessarily ignoring hints. Would you be ok with naming the config `table.optimizer.query-hints.enabled`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33168) An error occurred when executing sql, java.lang.NoSuchFieldError: operands
[
https://issues.apache.org/jira/browse/FLINK-33168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
macdoor615 updated FLINK-33168:
---
Description:
Environment:
{code:java}
Linux hb3-prod-hadoop-006 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Sep 21
06:49:25 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
openjdk version "1.8.0_382"
OpenJDK Runtime Environment (build 1.8.0_382-b05)
OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode)
flink-1.18.0-RC1 ,
https://github.com/apache/flink/releases/tag/release-1.18.0-rc1
{code}
I execute the following sql in sql-client.sh.
{code:java}
insert into svc1_paimon_prod.cq.b_customer_ecus
select
rcus.id id,
if(cus.id is not null, cus.id, try_cast(NULL as string)) cus_id,
if(cus.id is null and cus_rownum = 1, rcus.id, try_cast(NULL as string))
newCus_id,
companyID,
customerProvinceNumber,
mobilePhone,
oprCode,
customerNum,
staffName,
location,
staffNumber,
extendInfo,
customerName,
case when companyID='000' then '名称1'
when companyID='002' then '名称2'
else '新名称'
end prov,
row (
accessToken,
busType,
cutOffDay,
domain,
envFlag,
routeType,
routeValue,
sessionID,
sign,
signMethod,
org_timeStamp,
transIDO,
userPartyID,
version
) raw_message,
named_struct(
'id', cus.id,
'name', cus.name,
'code', cus.code,
'customerlevel', cus.customerlevel,
'prov', cus.prov,
'container', cus.container,
'crtime', cus.crtime,
'updtime', cus.updtime
) existing_cus,
cus_rownum,
to_timestamp(org_timeStamp, 'MMddHHmmss') as org_timeStamp,
raw_rowtime,
localtimestamp as raw_rowtime1,
dt
from svc1_paimon_prod.raw_data.abscustinfoserv_content_append_cq
/*+ OPTIONS('consumer-id' = '创建新客户id') */
rcus
left join svc1_mysql_test.gem_svc1_vpn.bv_customer
FOR SYSTEM_TIME AS OF rcus.proctime AS cus on rcus.customerNum=cus.code
{code}
There are the following jar files in the flink/lib directory.
{code:java}
commons-cli-1.5.0.jar
flink-cep-1.18.0.jar
flink-connector-files-1.18.0.jar
flink-connector-jdbc-3.1.1-1.17.jar
flink-csv-1.18.0.jar
flink-dist-1.18.0.jar
flink-json-1.18.0.jar
flink-orc-1.18.0.jar
flink-parquet-1.18.0.jar
flink-scala_2.12-1.18.0.jar
flink-sql-avro-1.18.0.jar
flink-sql-avro-confluent-registry-1.18.0.jar
flink-sql-connector-elasticsearch7-3.0.0-1.16.jar
flink-sql-connector-hive-3.1.3_2.12-1.18.0.jar
flink-sql-connector-kafka-3.0.0-1.17.jar
flink-sql-orc-1.18.0.jar
flink-sql-parquet-1.18.0.jar
flink-table-api-java-uber-1.18.0.jar
flink-table-api-scala_2.12-1.18.0.jar
flink-table-api-scala-bridge_2.12-1.18.0.jar
flink-table-planner_2.12-1.18.0.jar
flink-table-runtime-1.18.0.jar
jline-reader-3.23.0.jar
jline-terminal-3.23.0.jar
kafka-clients-3.5.1.jar
log4j-1.2-api-2.17.1.jar
log4j-api-2.17.1.jar
log4j-core-2.17.1.jar
log4j-slf4j-impl-2.17.1.jar
mysql-connector-j-8.1.0.jar
paimon-flink-1.18-0.6-20230929.002044-11.jar{code}
Works correctly in version 1.17.1, but produces the following error in
1.18.0-RC1
{code:java}
2023-09-29 14:04:11,438 ERROR
org.apache.flink.table.gateway.service.operation.OperationManager [] - Failed
to execute the operation fe1b0a58-b822-49c0-b1ae-ce73d16f92da.
java.lang.NoSuchFieldError: operands
at org.apache.calcite.plan.RelOptRule.operand(RelOptRule.java:129)
~[flink-sql-connector-hive-3.1.3_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.logical.SimplifyFilterConditionRule.(SimplifyFilterConditionRule.scala:36)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.logical.SimplifyFilterConditionRule$.(SimplifyFilterConditionRule.scala:94)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.logical.SimplifyFilterConditionRule$.(SimplifyFilterConditionRule.scala)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.FlinkStreamRuleSets$.(FlinkStreamRuleSets.scala:35)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.FlinkStreamRuleSets$.(FlinkStreamRuleSets.scala)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.optimize.program.FlinkStreamProgram$.buildProgram(FlinkStreamProgram.scala:57)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.$anonfun$optimizeTree$1(StreamCommonSubGraphBasedOptimizer.scala:169)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at scala.Option.getOrElse(Option.scala:189)
~[flink-scala_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:169)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
[jira] [Created] (FLINK-33168) An error occurred when executing sql, java.lang.NoSuchFieldError: operands
macdoor615 created FLINK-33168:
--
Summary: An error occurred when executing sql,
java.lang.NoSuchFieldError: operands
Key: FLINK-33168
URL: https://issues.apache.org/jira/browse/FLINK-33168
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.18.0
Reporter: macdoor615
Environment:
{code:java}
Linux hb3-prod-hadoop-006 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Sep 21
06:49:25 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
openjdk version "1.8.0_382"
OpenJDK Runtime Environment (build 1.8.0_382-b05)
OpenJDK 64-Bit Server VM (build 25.382-b05, mixed mode)
flink-1.18.0-RC1 ,
https://github.com/apache/flink/releases/tag/release-1.18.0-rc1
{code}
I execute the following sql in sql-client.sh.
{code:java}
insert into svc1_paimon_prod.cq.b_customer_ecus
select
rcus.id id,
if(cus.id is not null, cus.id, try_cast(NULL as string)) cus_id,
if(cus.id is null and cus_rownum = 1, rcus.id, try_cast(NULL as string))
newCus_id,
companyID,
customerProvinceNumber,
mobilePhone,
oprCode,
customerNum,
staffName,
location,
staffNumber,
extendInfo,
customerName,
case when companyID='000' then '名称1'
when companyID='002' then '名称2'
else '新名称'
end prov,
row (
accessToken,
busType,
cutOffDay,
domain,
envFlag,
routeType,
routeValue,
sessionID,
sign,
signMethod,
org_timeStamp,
transIDO,
userPartyID,
version
) raw_message,
named_struct(
'id', cus.id,
'name', cus.name,
'code', cus.code,
'customerlevel', cus.customerlevel,
'prov', cus.prov,
'container', cus.container,
'crtime', cus.crtime,
'updtime', cus.updtime
) existing_cus,
cus_rownum,
to_timestamp(org_timeStamp, 'MMddHHmmss') as org_timeStamp,
raw_rowtime,
localtimestamp as raw_rowtime1,
dt
from svc1_paimon_{job_env}.raw_data.abscustinfoserv_content_append_cq
/*+ OPTIONS('consumer-id' = '创建新客户id') */
rcus
left join svc1_mysql_test.gem_svc1_vpn.bv_customer
FOR SYSTEM_TIME AS OF rcus.proctime AS cus on rcus.customerNum=cus.code
{code}
There are the following jar files in the flink/lib directory.
{code:java}
commons-cli-1.5.0.jar
flink-cep-1.18.0.jar
flink-connector-files-1.18.0.jar
flink-connector-jdbc-3.1.1-1.17.jar
flink-csv-1.18.0.jar
flink-dist-1.18.0.jar
flink-json-1.18.0.jar
flink-orc-1.18.0.jar
flink-parquet-1.18.0.jar
flink-scala_2.12-1.18.0.jar
flink-sql-avro-1.18.0.jar
flink-sql-avro-confluent-registry-1.18.0.jar
flink-sql-connector-elasticsearch7-3.0.0-1.16.jar
flink-sql-connector-hive-3.1.3_2.12-1.18.0.jar
flink-sql-connector-kafka-3.0.0-1.17.jar
flink-sql-orc-1.18.0.jar
flink-sql-parquet-1.18.0.jar
flink-table-api-java-uber-1.18.0.jar
flink-table-api-scala_2.12-1.18.0.jar
flink-table-api-scala-bridge_2.12-1.18.0.jar
flink-table-planner_2.12-1.18.0.jar
flink-table-runtime-1.18.0.jar
jline-reader-3.23.0.jar
jline-terminal-3.23.0.jar
kafka-clients-3.5.1.jar
log4j-1.2-api-2.17.1.jar
log4j-api-2.17.1.jar
log4j-core-2.17.1.jar
log4j-slf4j-impl-2.17.1.jar
mysql-connector-j-8.1.0.jar
paimon-flink-1.18-0.6-20230929.002044-11.jar{code}
Works correctly in version 1.17.1, but produces the following error in
1.18.0-RC1
{code:java}
2023-09-29 14:04:11,438 ERROR
org.apache.flink.table.gateway.service.operation.OperationManager [] - Failed
to execute the operation fe1b0a58-b822-49c0-b1ae-ce73d16f92da.
java.lang.NoSuchFieldError: operands
at org.apache.calcite.plan.RelOptRule.operand(RelOptRule.java:129)
~[flink-sql-connector-hive-3.1.3_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.logical.SimplifyFilterConditionRule.(SimplifyFilterConditionRule.scala:36)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.logical.SimplifyFilterConditionRule$.(SimplifyFilterConditionRule.scala:94)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.logical.SimplifyFilterConditionRule$.(SimplifyFilterConditionRule.scala)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.FlinkStreamRuleSets$.(FlinkStreamRuleSets.scala:35)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.rules.FlinkStreamRuleSets$.(FlinkStreamRuleSets.scala)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.optimize.program.FlinkStreamProgram$.buildProgram(FlinkStreamProgram.scala:57)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at
org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.$anonfun$optimizeTree$1(StreamCommonSubGraphBasedOptimizer.scala:169)
~[flink-table-planner_2.12-1.18.0.jar:1.18.0]
at scala.Option.getOrElse(Option.scala:189)
~[flink-scala_2.12-1.18.0.jar:1.18.0]
at
