haf commented on issue #6594: [FLINK-9311] [pubsub] Added PubSub source connector with support for checkpointing (ATLEAST_ONCE) URL: https://github.com/apache/flink/pull/6594#issuecomment-484884659 I'm testing this source on spotty connections (as I do most of my work on the go), and when streaming a lot of messages over 4G, at some point, the source logs this: ``` 14:40:33,649 INFO io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - {} ignoring {} frame for stream {} {} 14:40:39,010 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 23 for job f2aed8990f452e357b8f15f6452e4244 (12951720 bytes in 18142 ms). 14:40:39,013 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 24 @ 1555677639010 for job f2aed8990f452e357b8f15f6452e4244. 14:41:11,608 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 24 for job f2aed8990f452e357b8f15f6452e4244 (14193479 bytes in 32593 ms). 14:41:11,614 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 25 @ 1555677671608 for job f2aed8990f452e357b8f15f6452e4244. 14:41:22,628 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 25 for job f2aed8990f452e357b8f15f6452e4244 (16366371 bytes in 11017 ms). 14:41:22,638 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 26 @ 1555677682628 for job f2aed8990f452e357b8f15f6452e4244. 14:41:43,287 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 26 for job f2aed8990f452e357b8f15f6452e4244 (17107164 bytes in 20655 ms). 14:41:43,290 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 27 @ 1555677703287 for job f2aed8990f452e357b8f15f6452e4244. 14:42:19,171 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Completed checkpoint 27 for job f2aed8990f452e357b8f15f6452e4244 (18502218 bytes in 35880 ms). 14:42:19,174 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Triggering checkpoint 28 @ 1555677739171 for job f2aed8990f452e357b8f15f6452e4244. 14:42:55,863 INFO io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - {} ignoring {} frame for stream {} {} 14:43:00,242 INFO io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - {} ignoring {} frame for stream {} {} 14:43:00,242 INFO io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder - {} ignoring {} frame for stream {} {} 1 ``` And it would seem no further events are being pushed down the streaming pipeline. Sometimes, I also get: ``` 14:43:23,816 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Request slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job 2d2461182e268f6838e957972a9a0361 with allocation id AllocationID{8f8a93b52291c2fa883df2fa1a7735de}. 14:43:23,816 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot for AllocationID{fc0b24a95227bdbfa46a4119fb907d19}. 14:43:23,816 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Add job 2d2461182e268f6838e957972a9a0361 for job leader monitoring. 14:43:23,816 INFO org.apache.flink.runtime.taskexecutor.JobLeaderService - Try to register at job manager akka://flink/user/jobmanager_1 with leader id 692682c0-30b2-4900-ad43-3d94acf9a2c6. 14:43:23,816 WARN org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService - Error notifying leader listener about new leader java.lang.IllegalStateException: The RPC connection is already closed at org.apache.flink.util.Preconditions.checkState(Preconditions.java:195) at org.apache.flink.runtime.registration.RegisteredRpcConnection.start(RegisteredRpcConnection.java:91) at org.apache.flink.runtime.taskexecutor.JobLeaderService$JobManagerLeaderListener.notifyLeaderAddress(JobLeaderService.java:327) at org.apache.flink.runtime.highavailability.nonha.embedded.EmbeddedLeaderService$NotifyOfLeaderCall.run(EmbeddedLeaderService.java:430) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ``` All of this is running locally on my laptop without ZK, so there should be no problems talking between JM and TMs.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
