[
https://issues.apache.org/jira/browse/FLINK-33251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840061#comment-17840061
]
Robert Metzger commented on FLINK-33251:
----------------------------------------
I'm having this problem with 1.19.0 as well, on a M1 MBP.
The problem is tricky to reproduce (e.g. it doesn't happen all the times)
{code}
2024-04-23 12:51:14,317 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Shutting down rest endpoint.
2024-04-23 12:51:14,317 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 2
thread-local buffer(s) from thread: flink-rest-client-netty-thread-1
2024-04-23 12:51:14,318 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest endpoint shutdown complete.
2024-04-23 12:51:14,318 TRACE
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop [] -
instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@c1d225b
2024-04-23 12:51:14,318 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest client endpoint started.
2024-04-23 12:51:14,318 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.job.coordination.ClientCoordinationRequestBody
to
localhost:8081/v1/jobs/5d8d1b8ef7dc49381a3855ae10a18ec5/coordinators/b728d985904d42b0fdd945a9e3253fca
2024-04-23 12:51:14,320 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Received response
{"serializedCoordinationResult":"rO0ABXNyAExvcmcuYXBhY2hlLmZsaW5rLnN0cmVhbWluZy5hcGkub3BlcmF0b3JzLmNvbGxlY3QuQ29sbGVjdENvb3JkaW5hdGlvblJlc3BvbnNlAAAAAAAAAAECAANKABZsYXN0Q2hlY2twb2ludGVkT2Zmc2V0TAARc2VyaWFsaXplZFJlc3VsdHN0ABBMamF2YS91dGlsL0xpc3Q7TAAHdmVyc2lvbnQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAABzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHh0ACQ3MzgyMjA5Ni0wODE2LTQ5NTMtODA4NC1kMDJhZTg0ZjNhNWU="}.
2024-04-23 12:51:14,321 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Shutting down rest endpoint.
2024-04-23 12:51:14,321 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 3
thread-local buffer(s) from thread: flink-rest-client-netty-thread-1
2024-04-23 12:51:14,321 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest endpoint shutdown complete.
2024-04-23 12:51:14,390 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.EmptyRequestBody to
localhost:62113/v2/sessions/88b98272-be33-4303-a649-942acd213e84/heartbeat
2024-04-23 12:51:14,391 TRACE org.apache.flink.runtime.rest.FileUploadHandler
[] - Received request.
URL:/v2/sessions/88b98272-be33-4303-a649-942acd213e84/heartbeat Method:POST
2024-04-23 12:51:14,391 TRACE
org.apache.flink.table.gateway.rest.handler.session.TriggerSessionHeartbeatHandler
[] - Received request
/v2/sessions/88b98272-be33-4303-a649-942acd213e84/heartbeat.
2024-04-23 12:51:14,391 TRACE
org.apache.flink.table.gateway.rest.handler.session.TriggerSessionHeartbeatHandler
[] - Starting request processing.
2024-04-23 12:51:14,391 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Received response {}.
2024-04-23 12:51:14,425 TRACE
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop [] -
instrumented a special java.util.Set into: sun.nio.ch.KQueueSelectorImpl@1436813
2024-04-23 12:51:14,425 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest client endpoint started.
2024-04-23 12:51:14,426 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.EmptyRequestBody to
localhost:8081/v1/jobs/5d8d1b8ef7dc49381a3855ae10a18ec5/status
2024-04-23 12:51:14,434 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Received response {"status":"RUNNING"}.
2024-04-23 12:51:14,435 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Shutting down rest endpoint.
2024-04-23 12:51:14,435 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 2
thread-local buffer(s) from thread: flink-rest-client-netty-thread-1
2024-04-23 12:51:14,435 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest endpoint shutdown complete.
2024-04-23 12:51:14,435 TRACE
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop [] -
instrumented a special java.util.Set into:
sun.nio.ch.KQueueSelectorImpl@64f0cb77
2024-04-23 12:51:14,435 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest client endpoint started.
2024-04-23 12:51:14,436 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.job.coordination.ClientCoordinationRequestBody
to
localhost:8081/v1/jobs/5d8d1b8ef7dc49381a3855ae10a18ec5/coordinators/b728d985904d42b0fdd945a9e3253fca
2024-04-23 12:51:21,111 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.EmptyRequestBody to
localhost:62113/v2/sessions/88b98272-be33-4303-a649-942acd213e84/operations/32015c0c-d543-48fc-bec5-2ec8f5ae59e1/close
2024-04-23 12:51:21,113 TRACE org.apache.flink.runtime.rest.FileUploadHandler
[] - Received request.
URL:/v2/sessions/88b98272-be33-4303-a649-942acd213e84/operations/32015c0c-d543-48fc-bec5-2ec8f5ae59e1/close
Method:DELETE
2024-04-23 12:51:21,113 TRACE
org.apache.flink.table.gateway.rest.handler.operation.CloseOperationHandler []
- Received request
/v2/sessions/88b98272-be33-4303-a649-942acd213e84/operations/32015c0c-d543-48fc-bec5-2ec8f5ae59e1/close.
2024-04-23 12:51:21,113 TRACE
org.apache.flink.table.gateway.rest.handler.operation.CloseOperationHandler []
- Starting request processing.
2024-04-23 12:51:21,113 DEBUG
org.apache.flink.table.gateway.service.operation.OperationManager [] - Convert
operation 32015c0c-d543-48fc-bec5-2ec8f5ae59e1 from FINISHED to CLOSED.
2024-04-23 12:51:21,114 TRACE
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop [] -
instrumented a special java.util.Set into:
sun.nio.ch.KQueueSelectorImpl@5679090f
2024-04-23 12:51:21,114 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest client endpoint started.
2024-04-23 12:51:21,114 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.EmptyRequestBody to
localhost:8081/v1/jobs/5d8d1b8ef7dc49381a3855ae10a18ec5/status
2024-04-23 12:51:21,115 WARN
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher [] - An
exception occurred when fetching query results
java.lang.InterruptedException: null
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386)
~[?:?]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:173)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:131)
[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:126)
[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:100)
[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.table.planner.connectors.CollectDynamicSink$CloseableRowIteratorWrapper.hasNext(CollectDynamicSink.java:247)
[flink-table-planner_833723f8-2501-4b26-9e61-bfe84f51b9ca.jar:1.19.0]
at
org.apache.flink.table.gateway.service.result.ResultStore$ResultRetrievalThread.run(ResultStore.java:155)
[flink-sql-gateway-1.19.0.jar:1.19.0]
2024-04-23 12:51:21,117 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Received response {"status":"RUNNING"}.
2024-04-23 12:51:21,117 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Shutting down rest endpoint.
2024-04-23 12:51:21,117 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 2
thread-local buffer(s) from thread: flink-rest-client-netty-thread-1
2024-04-23 12:51:21,117 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest endpoint shutdown complete.
2024-04-23 12:51:21,118 TRACE
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop [] -
instrumented a special java.util.Set into:
sun.nio.ch.KQueueSelectorImpl@7de456e9
2024-04-23 12:51:21,118 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest client endpoint started.
2024-04-23 12:51:21,119 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.EmptyRequestBody to
localhost:8081/v1/jobs/5d8d1b8ef7dc49381a3855ae10a18ec5?mode=cancel
2024-04-23 12:51:21,119 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Received response {"status":"CLOSED"}.
2024-04-23 12:51:21,125 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Received response {}.
2024-04-23 12:51:21,125 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Shutting down rest endpoint.
2024-04-23 12:51:21,125 DEBUG
org.apache.flink.shaded.netty4.io.netty.buffer.PoolThreadCache [] - Freed 2
thread-local buffer(s) from thread: flink-rest-client-netty-thread-1
2024-04-23 12:51:21,125 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Rest endpoint shutdown complete.
2024-04-23 12:51:21,140 WARN org.apache.flink.table.client.cli.CliClient
[] - Could not execute SQL statement.
org.apache.flink.table.client.gateway.SqlExecutionException: Error while
retrieving result.
at
org.apache.flink.table.client.gateway.result.CollectResultBase$ResultRetrievalThread.run(CollectResultBase.java:84)
~[flink-sql-client-1.19.0.jar:1.19.0]
Caused by: org.apache.flink.table.client.gateway.SqlExecutionException: Failed
to get response for the operation 32015c0c-d543-48fc-bec5-2ec8f5ae59e1.
at
org.apache.flink.table.client.gateway.ExecutorImpl.getFetchResultResponse(ExecutorImpl.java:489)
~[flink-sql-client-1.19.0.jar:1.19.0]
at
org.apache.flink.table.client.gateway.ExecutorImpl.access$200(ExecutorImpl.java:108)
~[flink-sql-client-1.19.0.jar:1.19.0]
at
org.apache.flink.table.client.gateway.ExecutorImpl$RowDataInfoIterator.fetchResults(ExecutorImpl.java:392)
~[flink-sql-client-1.19.0.jar:1.19.0]
at
org.apache.flink.table.client.gateway.ExecutorImpl$RowDataInfoIterator.hasNext(ExecutorImpl.java:376)
~[flink-sql-client-1.19.0.jar:1.19.0]
at
org.apache.flink.table.client.gateway.StatementResult.hasNext(StatementResult.java:106)
~[flink-sql-client-1.19.0.jar:1.19.0]
at
org.apache.flink.table.client.gateway.result.CollectResultBase$ResultRetrievalThread.run(CollectResultBase.java:80)
~[flink-sql-client-1.19.0.jar:1.19.0]
Caused by:
org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException:
connection timed out: localhost/127.0.0.1:62113
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[flink-dist-1.19.0.jar:1.19.0]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-04-23 12:51:21,141 DEBUG org.apache.flink.runtime.rest.RestClient
[] - Sending request of class class
org.apache.flink.runtime.rest.messages.EmptyRequestBody to
localhost:62113/v2/sessions/88b98272-be33-4303-a649-942acd213e84
2024-04-23 12:51:21,142 TRACE org.apache.flink.runtime.rest.FileUploadHandler
[] - Received request.
URL:/v2/sessions/88b98272-be33-4303-a649-942acd213e84 Method:GET
2024-04-23 12:51:21,142 TRACE
org.apache.flink.table.gateway.rest.handler.session.GetSessionConfigHandler []
- Received request /v2/sessions/88b98272-be33-4303-a649-942acd213e84.
2024-04-23 12:51:21,143 TRACE
org.apache.flink.table.gateway.rest.handler.session.GetSessionConfigHandler []
- Starting request processing.
{code}
> SQL Client query execution aborts after a few seconds: ConnectTimeoutException
> ------------------------------------------------------------------------------
>
> Key: FLINK-33251
> URL: https://issues.apache.org/jira/browse/FLINK-33251
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
> Affects Versions: 1.18.0, 1.17.1
> Environment: Macbook Pro
> Apple M1 Max
>
> {code:java}
> $ uname -a
> Darwin asgard08 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:43 PDT
> 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6000 arm64
> {code}
> {code:bash}
> $ java --version
> openjdk 11.0.20.1 2023-08-24
> OpenJDK Runtime Environment Homebrew (build 11.0.20.1+0)
> OpenJDK 64-Bit Server VM Homebrew (build 11.0.20.1+0, mixed mode)
> $ mvn --version
> Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546)
> Maven home: /opt/homebrew/Cellar/maven/3.9.5/libexec
> Java version: 11.0.20.1, vendor: Homebrew, runtime:
> /opt/homebrew/Cellar/openjdk@11/11.0.20.1/libexec/openjdk.jdk/Contents/Home
> Default locale: en_GB, platform encoding: UTF-8
> OS name: "mac os x", version: "14.0", arch: "aarch64", family: "mac"
> {code}
> Reporter: Robin Moffatt
> Priority: Major
> Attachments: log.zip
>
>
> If I run a streaming query from an unbounded connector from the SQL Client,
> it bombs out after ~15 seconds.
> {code:java}
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException:
> connection timed out: localhost/127.0.0.1:52596
> {code}
> This *doesn't* happen on 1.16.2. It *does* happen on *1.17.1* and *1.18* that
> I have just built locally (git repo hash `9b837727b6d`).
> The corresponding task's status in the Web UI shows as `CANCELED`.
> ---
> h2. To reproduce
> Launch local cluster and SQL client
> {code}
> ➜ flink-1.18-SNAPSHOT ./bin/start-cluster.sh
> Starting cluster.
> Starting standalonesession daemon on host asgard08.
> Starting taskexecutor daemon on host asgard08.
> ➜ flink-1.18-SNAPSHOT ./bin/sql-client.sh
> […]
> Flink SQL>
> {code}
> Set streaming mode and result mode
> {code:sql}
> Flink SQL> SET 'execution.runtime-mode' = 'STREAMING';
> [INFO] Execute statement succeed.
> Flink SQL> SET 'sql-client.execution.result-mode' = 'changelog';
> [INFO] Execute statement succeed.
> {code}
> Define a table to read data from CSV files in a folder
> {code:sql}
> CREATE TABLE firewall (
> event_time STRING,
> source_ip STRING,
> dest_ip STRING,
> source_prt INT,
> dest_prt INT
> ) WITH (
> 'connector' = 'filesystem',
> 'path' = 'file:///tmp/firewall/',
> 'format' = 'csv',
> 'source.monitor-interval' = '1' -- unclear from the docs what the unit is
> here
> );
> {code}
> Create a CSV file to read in
> {code:bash}
> $ mkdir /tmp/firewall
> $ cat > /tmp/firewall/data.csv <<EOF
> 2018-05-11 00:19:34,151.35.34.162,125.26.20.222,2014,68
> 2018-05-11 22:20:43,114.24.126.190,21.68.21.69,379,1619
> EOF
> {code}
> Run a streaming query
> {code}
> SELECT * FROM firewall;
> {code}
> You will get results showing (and if you add another data file it will show
> up) - but after ~30 seconds the query aborts and throws an error back to the
> user at the SQL Client prompt
> {code}
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.shaded.netty4.io.netty.channel.ConnectTimeoutException:
> connection timed out: localhost/127.0.0.1:58470
> Flink SQL>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
