[
https://issues.apache.org/jira/browse/FLINK-29755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17636424#comment-17636424
]
Leonard Xu edited comment on FLINK-29755 at 11/22/22 7:48 AM:
--------------------------------------------------------------
[~syhily] I also found some shade error log and pulsar internal error log,
these error logs really make the troubleshoot harder, could you take a look ?
{noformat}
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: 2022-11-19 03:05:19,977 ERROR
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.rejectedExecution
[] - Failed to submit a listener notification task. Event loop shut down?
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: java.lang.NoClassDefFoundError:
org/apache/pulsar/shade/io/netty/util/concurrent/GlobalEventExecutor$2
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:223)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.execute0(GlobalEventExecutor.java:211)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:205)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1057)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at java.lang.Thread.run(Thread.java:750)
[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: Caused by: java.lang.ClassNotFoundException:
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor$2
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: ... 12 more{noformat}
{noformat}
ERROR org.apache.pulsar.broker.service.ServerCnx - Send response error for
END_TXN request 2213444997852929784.
ERROR org.apache.flink.shaded.curator5.org.apache.curator.ConnectionState [] -
Authentication failed
{noformat}
was (Author: leonard xu):
[~syhily] I also found some shade error log and pulsar internal error log,
these error logs really make the troubleshoot harder, could you take a look ?
{noformat}
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: 2022-11-19 03:05:19,977 ERROR
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.rejectedExecution
[] - Failed to submit a listener notification task. Event loop shut down?
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: java.lang.NoClassDefFoundError:
org/apache/pulsar/shade/io/netty/util/concurrent/GlobalEventExecutor$2
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:223)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.execute0(GlobalEventExecutor.java:211)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:205)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1057)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at java.lang.Thread.run(Thread.java:750)
[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: Caused by: java.lang.ClassNotFoundException:
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor$2
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:67)
~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.flink.util.ChildFirstClassLoader.loadClassWithoutExceptionHandling(ChildFirstClassLoader.java:74)
~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:51)
~[flink-dist-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: at
java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_342]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: ... 12 more{noformat}
{noformat}
ERROR org.apache.pulsar.broker.service.ServerCnx - Send response error for
END_TXN request 2213444997852929784.
ERROR org.apache.flink.shaded.curator5.org.apache.curator.ConnectionState [] -
Authentication failed
{noformat}
> PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing
> TaskManagers
> ---------------------------------------------------------------------------------
>
> Key: FLINK-29755
> URL: https://issues.apache.org/jira/browse/FLINK-29755
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
> Affects Versions: 1.16.0, 1.17.0
> Reporter: Matthias Pohl
> Priority: Critical
> Labels: test-stability
> Attachments: PulsarSourceUnorderedE2ECase.testSavepoint.log
>
>
> [This
> build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42325&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=13932]
> failed (not exclusively) due to a problem with
> {{PulsarSourceUnorderedE2ECase.testSavepoint}}. It seems like there were no
> TaskManagers spun up which resulted in the test job failing with a
> {{NoResourceAvailableException}}.
> {code}
> org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPoolBridge [] -
> Could not acquire the minimum required resources, failing slot requests.
> Acquired: []. Current slot pool status: Registered TMs: 0, registered slots:
> 0 free slots: 0
> {code}
> I didn't raise this one to critical because it looks like a missing
> TaskManager test environment issue. I attached the e2e test-specific logs to
> the Jira issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
