[ https://issues.apache.org/jira/browse/KAFKA-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17279097#comment-17279097 ]
prabhu biradar commented on KAFKA-12279: ---------------------------------------- Hi Sophie Blee-Goldman, 1) This error occurs on every restart post 2.7 Kafka upgrade. 2) Yes we are running 12 instances (6 ECS TASK and each TASKk runs 2 threads) 3) Below is the detailed logs starting from application restart. Completed 23.2 KiB/23.2 KiB (228.6 KiB/s) with 1 file(s) remainingCompleted 23.2 KiB/23.2 KiB (228.6 KiB/s) with 1 file(s) remainingdownload: s3://x/perf/kafka-client-truststore.jks to ./client.truststore.jksCompleted 256.0 KiB/20.9 MiB (1.9 MiB/s) with 1 file(s) remainingCompleted 512.0 KiB/20.9 MiB (3.7 MiB/s) with 1 file(s) remainingCompleted 768.0 KiB/20.9 MiB (5.4 MiB/s) with 1 file(s) remainingCompleted 1.0 MiB/20.9 MiB (7.2 MiB/s) with 1 file(s) remaining Completed 1.2 MiB/20.9 MiB (8.9 MiB/s) with 1 file(s) remaining Completed 1.5 MiB/20.9 MiB (10.5 MiB/s) with 1 file(s) remaining Completed 1.8 MiB/20.9 MiB (12.1 MiB/s) with 1 file(s) remaining Completed 2.0 MiB/20.9 MiB (13.8 MiB/s) with 1 file(s) remaining Completed 2.2 MiB/20.9 MiB (15.3 MiB/s) with 1 file(s) remaining Completed 2.5 MiB/20.9 MiB (16.9 MiB/s) with 1 file(s) remaining Completed 2.8 MiB/20.9 MiB (18.3 MiB/s) with 1 file(s) remaining Completed 3.0 MiB/20.9 MiB (19.9 MiB/s) with 1 file(s) remaining Completed 3.2 MiB/20.9 MiB (21.4 MiB/s) with 1 file(s) remaining Completed 3.5 MiB/20.9 MiB (22.8 MiB/s) with 1 file(s) remaining Completed 3.8 MiB/20.9 MiB (24.3 MiB/s) with 1 file(s) remaining Completed 4.0 MiB/20.9 MiB (25.8 MiB/s) with 1 file(s) remaining Completed 4.2 MiB/20.9 MiB (27.0 MiB/s) with 1 file(s) remaining Completed 4.5 MiB/20.9 MiB (28.4 MiB/s) with 1 file(s) remaining Completed 4.8 MiB/20.9 MiB (29.9 MiB/s) with 1 file(s) remaining Completed 5.0 MiB/20.9 MiB (31.1 MiB/s) with 1 file(s) remaining Completed 5.2 MiB/20.9 MiB (32.4 MiB/s) with 1 file(s) remaining Completed 5.5 MiB/20.9 MiB (33.7 MiB/s) with 1 file(s) remaining Completed 5.8 MiB/20.9 MiB (34.9 MiB/s) with 1 file(s) remaining Completed 6.0 MiB/20.9 MiB (36.1 MiB/s) with 1 file(s) remaining Completed 6.2 MiB/20.9 MiB (37.4 MiB/s) with 1 file(s) remaining Completed 6.5 MiB/20.9 MiB (38.7 MiB/s) with 1 file(s) remaining Completed 6.8 MiB/20.9 MiB (39.9 MiB/s) with 1 file(s) remaining Completed 7.0 MiB/20.9 MiB (41.1 MiB/s) with 1 file(s) remaining Completed 7.2 MiB/20.9 MiB (42.3 MiB/s) with 1 file(s) remaining Completed 7.5 MiB/20.9 MiB (43.5 MiB/s) with 1 file(s) remaining Completed 7.8 MiB/20.9 MiB (44.6 MiB/s) with 1 file(s) remaining Completed 8.0 MiB/20.9 MiB (45.7 MiB/s) with 1 file(s) remaining Completed 8.2 MiB/20.9 MiB (47.1 MiB/s) with 1 file(s) remaining Completed 8.5 MiB/20.9 MiB (48.1 MiB/s) with 1 file(s) remaining Completed 8.8 MiB/20.9 MiB (49.1 MiB/s) with 1 file(s) remaining Completed 9.0 MiB/20.9 MiB (50.2 MiB/s) with 1 file(s) remaining Completed 9.2 MiB/20.9 MiB (51.5 MiB/s) with 1 file(s) remaining Completed 9.5 MiB/20.9 MiB (52.3 MiB/s) with 1 file(s) remaining Completed 9.8 MiB/20.9 MiB (53.4 MiB/s) with 1 file(s) remaining Completed 10.0 MiB/20.9 MiB (54.5 MiB/s) with 1 file(s) remainingCompleted 10.2 MiB/20.9 MiB (55.3 MiB/s) with 1 file(s) remainingCompleted 10.5 MiB/20.9 MiB (56.3 MiB/s) with 1 file(s) remainingCompleted 10.8 MiB/20.9 MiB (57.4 MiB/s) with 1 file(s) remainingCompleted 11.0 MiB/20.9 MiB (58.3 MiB/s) with 1 file(s) remainingCompleted 11.2 MiB/20.9 MiB (59.3 MiB/s) with 1 file(s) remainingCompleted 11.5 MiB/20.9 MiB (60.2 MiB/s) with 1 file(s) remainingCompleted 11.8 MiB/20.9 MiB (61.1 MiB/s) with 1 file(s) remainingCompleted 12.0 MiB/20.9 MiB (62.0 MiB/s) with 1 file(s) remainingCompleted 12.2 MiB/20.9 MiB (63.0 MiB/s) with 1 file(s) remainingCompleted 12.5 MiB/20.9 MiB (63.9 MiB/s) with 1 file(s) remainingCompleted 12.8 MiB/20.9 MiB (64.7 MiB/s) with 1 file(s) remainingCompleted 13.0 MiB/20.9 MiB (65.7 MiB/s) with 1 file(s) remainingCompleted 13.2 MiB/20.9 MiB (66.5 MiB/s) with 1 file(s) remainingCompleted 13.5 MiB/20.9 MiB (67.4 MiB/s) with 1 file(s) remainingCompleted 13.8 MiB/20.9 MiB (68.3 MiB/s) with 1 file(s) remainingCompleted 14.0 MiB/20.9 MiB (69.2 MiB/s) with 1 file(s) remainingCompleted 14.2 MiB/20.9 MiB (70.0 MiB/s) with 1 file(s) remainingCompleted 14.5 MiB/20.9 MiB (71.0 MiB/s) with 1 file(s) remainingCompleted 14.8 MiB/20.9 MiB (71.7 MiB/s) with 1 file(s) remainingCompleted 15.0 MiB/20.9 MiB (72.5 MiB/s) with 1 file(s) remainingCompleted 15.2 MiB/20.9 MiB (73.5 MiB/s) with 1 file(s) remainingCompleted 15.5 MiB/20.9 MiB (74.2 MiB/s) with 1 file(s) remainingCompleted 15.8 MiB/20.9 MiB (75.1 MiB/s) with 1 file(s) remainingCompleted 16.0 MiB/20.9 MiB (76.1 MiB/s) with 1 file(s) remainingCompleted 16.1 MiB/20.9 MiB (76.0 MiB/s) with 1 file(s) remainingCompleted 16.4 MiB/20.9 MiB (76.9 MiB/s) with 1 file(s) remainingCompleted 16.6 MiB/20.9 MiB (77.8 MiB/s) with 1 file(s) remainingCompleted 16.9 MiB/20.9 MiB (78.4 MiB/s) with 1 file(s) remainingCompleted 17.1 MiB/20.9 MiB (79.3 MiB/s) with 1 file(s) remainingCompleted 17.4 MiB/20.9 MiB (80.0 MiB/s) with 1 file(s) remainingCompleted 17.6 MiB/20.9 MiB (80.7 MiB/s) with 1 file(s) remainingCompleted 17.9 MiB/20.9 MiB (81.5 MiB/s) with 1 file(s) remainingCompleted 18.1 MiB/20.9 MiB (82.3 MiB/s) with 1 file(s) remainingCompleted 18.4 MiB/20.9 MiB (83.1 MiB/s) with 1 file(s) remainingCompleted 18.6 MiB/20.9 MiB (83.9 MiB/s) with 1 file(s) remainingCompleted 18.9 MiB/20.9 MiB (84.6 MiB/s) with 1 file(s) remainingCompleted 19.1 MiB/20.9 MiB (85.4 MiB/s) with 1 file(s) remainingCompleted 19.4 MiB/20.9 MiB (86.1 MiB/s) with 1 file(s) remainingCompleted 19.6 MiB/20.9 MiB (86.9 MiB/s) with 1 file(s) remainingCompleted 19.9 MiB/20.9 MiB (87.6 MiB/s) with 1 file(s) remainingCompleted 20.1 MiB/20.9 MiB (88.3 MiB/s) with 1 file(s) remainingCompleted 20.4 MiB/20.9 MiB (89.0 MiB/s) with 1 file(s) remainingCompleted 20.6 MiB/20.9 MiB (89.7 MiB/s) with 1 file(s) remainingCompleted 20.9 MiB/20.9 MiB (90.5 MiB/s) with 1 file(s) remainingdownload: s3://x-monitoring-us-east-1/AppServerAgent-4.5.17.28908.zip to ./AppServerAgent-4.5.17.28908.zipArchive: AppServerAgent-4.5.17.28908.zip creating: appdynamics/ inflating: appdynamics/javaagent.jar creating: appdynamics/multi-release/ inflating: appdynamics/multi-release/java9-impl.jar.asc inflating: appdynamics/multi-release/java9-impl.jar inflating: appdynamics/.DS_Store creating: appdynamics/ver20.6.0.30246/ inflating: appdynamics/ver20.6.0.30246/javaagent.jar creating: appdynamics/ver20.6.0.30246/multi-release/ inflating: appdynamics/ver20.6.0.30246/multi-release/java9-impl.jar.asc inflating: appdynamics/ver20.6.0.30246/multi-release/java9-impl.jar creating: appdynamics/ver20.6.0.30246/utils/ creating: appdynamics/ver20.6.0.30246/utils/keystorereader/ inflating: appdynamics/ver20.6.0.30246/utils/keystorereader/kr.jar inflating: appdynamics/ver20.6.0.30246/utils/keystorereader/kr.jar.asc creating: appdynamics/ver20.6.0.30246/utils/scs/ inflating: appdynamics/ver20.6.0.30246/utils/scs/scs-tool.jar.asc inflating: appdynamics/ver20.6.0.30246/utils/scs/scs-tool.jar creating: appdynamics/ver20.6.0.30246/utils/verifier/ inflating: appdynamics/ver20.6.0.30246/utils/verifier/jar-verifier.jar inflating: appdynamics/ver20.6.0.30246/utils/verifier/jar-verifier.jar.asc creating: appdynamics/ver20.6.0.30246/external-services/ creating: appdynamics/ver20.6.0.30246/external-services/netviz/ inflating: appdynamics/ver20.6.0.30246/external-services/netviz/netviz-service.properties.asc inflating: appdynamics/ver20.6.0.30246/external-services/netviz/netviz-service.properties inflating: appdynamics/ver20.6.0.30246/external-services/netviz/netviz-adapted.jar.asc creating: appdynamics/ver20.6.0.30246/external-services/netviz/lib/ inflating: appdynamics/ver20.6.0.30246/external-services/netviz/lib/protobuf-java-3.11.4.jar inflating: appdynamics/ver20.6.0.30246/external-services/netviz/lib/jeromq-0.3.5.jar.asc inflating: appdynamics/ver20.6.0.30246/external-services/netviz/lib/protobuf-java-3.11.4.jar.asc inflating: appdynamics/ver20.6.0.30246/external-services/netviz/lib/jeromq-0.3.5.jar inflating: appdynamics/ver20.6.0.30246/external-services/netviz/netviz-adapted.jar creating: appdynamics/ver20.6.0.30246/external-services/analytics/ inflating: appdynamics/ver20.6.0.30246/external-services/analytics/analytics-dynamic-service.jar.asc inflating: appdynamics/ver20.6.0.30246/external-services/analytics/analytics-dynamic-service.jar creating: appdynamics/ver20.6.0.30246/logs/ creating: appdynamics/ver20.6.0.30246/sdk-plugins/ creating: appdynamics/ver20.6.0.30246/sdk/ creating: appdynamics/ver20.6.0.30246/sdk/src/ creating: appdynamics/ver20.6.0.30246/sdk/src/java/ creating: appdynamics/ver20.6.0.30246/sdk/src/java/com/ creating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/ creating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/ creating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/ creating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/async/ inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/async/IAsyncServer.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/async/AsyncServer.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/async/IAsyncServer.java.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/async/AsyncClient.java.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/async/AsyncClient.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/async/AsyncServer.java.asc creating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/sync/ inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/sync/SyncClient.java.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/sync/ISyncServer.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/sync/SyncServer.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/sync/SyncClient.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/sync/SyncServer.java.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/exitcall/sync/ISyncServer.java.asc creating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/multithread/ inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/multithread/MultiThreadServer.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/multithread/MultiThreadedExecutor.java inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/multithread/MultiThreadedExecutor.java.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/java/com/appdynamics/sample/multithread/MultiThreadServer.java.asc creating: appdynamics/ver20.6.0.30246/sdk/src/docs/ creating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/ creating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/ creating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/ creating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/ creating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/async/ inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/async/AsyncServer.xhtml.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/async/AsyncClient.xhtml inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/async/AsyncServer.xhtml inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/async/IAsyncServer.xhtml.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/async/AsyncClient.xhtml.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/async/IAsyncServer.xhtml creating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/sync/ inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/sync/SyncClient.xhtml.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/sync/SyncClient.xhtml inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/sync/ISyncServer.xhtml.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/sync/ISyncServer.xhtml inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/sync/SyncServer.xhtml inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/exitcall/sync/SyncServer.xhtml.asc creating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/multithread/ inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/multithread/MutiThreadedExecutor.xhtml.asc inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/multithread/MutiThreadedExecutor.xhtml inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/multithread/MultiThreadServer.xhtml inflating: appdynamics/ver20.6.0.30246/sdk/src/docs/com/appdynamics/sample/multithread/MultiThreadServer.xhtml.asc creating: appdynamics/ver20.6.0.30246/lib/ inflating: appdynamics/ver20.6.0.30246/lib/common-utils.jar inflating: appdynamics/ver20.6.0.30246/lib/common-utils.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/appagent.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/singularity-log4j.jar inflating: appdynamics/ver20.6.0.30246/lib/appagent-boot.jar inflating: appdynamics/ver20.6.0.30246/lib/appagent.jar creating: appdynamics/ver20.6.0.30246/lib/tp/ inflating: appdynamics/ver20.6.0.30246/lib/tp/httpcore-nio-4.4.13.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/slf4j-api-1.6.6.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/xercesImpl-2.12.0.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/log4j-jcl-2.11.2.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-io-2.5.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-math-2.1.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-logging-1.2.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/javax.servlet-api-3.0.1.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-math-2.1.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/slf4j-api-1.6.6.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/asm.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-codec-1.11.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/httpcore-4.4.13.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/httpcore-4.4.13.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/dagger-2.4.2.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/log4j-api-2.11.2.3.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/disruptor-2.11.2.3.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/zt-zip-1.13.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/httpmime-4.5.12.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/scs-api.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/httpclient-cache-4.5.12.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-fileupload-1.4.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/log4j-jcl-2.11.2.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/asm.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/httpclient-cache-4.5.12.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/javax.inject-1.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/disruptor-2.11.2.3.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/log4j-core-2.11.2.3.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-io-2.5.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/log4j-api-2.11.2.3.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-codec-1.11.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/log4j-core-2.11.2.3.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/httpcore-nio-4.4.13.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/xercesImpl-2.12.0.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/singularity-common-1.0.1.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/xml-apis-1.4.01.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/javax.servlet-api-3.0.1.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/singularity-common-1.0.1.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/scs-api.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/dagger-2.4.2.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-fileupload-1.4.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/singularity-xml-1.0.2.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/httpclient-4.5.12.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/singularity-xml-1.0.2.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/httpclient-4.5.12.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/zt-zip-1.13.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/xml-apis-1.4.01.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/tp/javax.inject-1.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/httpmime-4.5.12.jar inflating: appdynamics/ver20.6.0.30246/lib/tp/commons-logging-1.2.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/singularity-log4j.jar.asc inflating: appdynamics/ver20.6.0.30246/lib/appagent-boot.jar.asc inflating: appdynamics/ver20.6.0.30246/readme.txt inflating: appdynamics/ver20.6.0.30246/readme.txt.asc creating: appdynamics/ver20.6.0.30246/conf/ inflating: appdynamics/ver20.6.0.30246/conf/transactions.xml inflating: appdynamics/ver20.6.0.30246/conf/service-endpoints.xml inflating: appdynamics/ver20.6.0.30246/conf/app-agent-config.xml.asc creating: appdynamics/ver20.6.0.30246/conf/jmx/ inflating: appdynamics/ver20.6.0.30246/conf/jmx/mbean-servers.dtd inflating: appdynamics/ver20.6.0.30246/conf/jmx/mbean-servers.dtd.asc inflating: appdynamics/ver20.6.0.30246/conf/jmx/mbean-servers.xsd.asc inflating: appdynamics/ver20.6.0.30246/conf/jmx/mbean-servers.xsd inflating: appdynamics/ver20.6.0.30246/conf/jmx/mbean-servers.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/jmx/mbean-servers.xml creating: appdynamics/ver20.6.0.30246/conf/jmx/servers/ inflating: appdynamics/ver20.6.0.30246/conf/jmx/servers/jmx-config.xsd inflating: appdynamics/ver20.6.0.30246/conf/jmx/servers/jmx-config.dtd inflating: appdynamics/ver20.6.0.30246/conf/jmx/servers/jmx-config.xsd.asc inflating: appdynamics/ver20.6.0.30246/conf/jmx/servers/jmx-config.dtd.asc inflating: appdynamics/ver20.6.0.30246/conf/app-agent-config.xml inflating: appdynamics/ver20.6.0.30246/conf/custom-activity-correlation.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/controller-info.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/transactions.xml.asc extracting: appdynamics/ver20.6.0.30246/conf/cacerts.jks inflating: appdynamics/ver20.6.0.30246/conf/service-endpoints.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/custom-activity-correlation.xml creating: appdynamics/ver20.6.0.30246/conf/logging/ inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j2.xml inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j-unknown.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j.dtd.asc inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j-unknown.xml inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j2-unknown.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j.xml inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j.dtd inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j2-unknown.xml inflating: appdynamics/ver20.6.0.30246/conf/logging/log4j2.xml.asc inflating: appdynamics/ver20.6.0.30246/conf/controller-info.xml inflating: appdynamics/ver20.6.0.30246/javaagent.jar.asc inflating: appdynamics/readme.txt inflating: appdynamics/readme.txt.asc creating: appdynamics/conf/ inflating: appdynamics/conf/controller-info.xml.asc extracting: appdynamics/conf/cacerts.jks inflating: appdynamics/conf/controller-info.xml inflating: appdynamics/javaagent.jar.asc /opt/appJAVA_AGENT_OPTS - " -javaagent:/opt/app/monitoring/appdynamics/javaagent.jar -Dappdynamics.agent.accountAccessKey=e596be0c-2f3e-4200-bddd-3714d4ec466d -Dappdynamics.agent.accountName=x-test -Dappdynamics.agent.applicationName=it-x-perf -Dappdynamics.controller.hostName=x-test.saas.appdynamics.com -Dappdynamics.controller.ssl.enabled=true -Dappdynamics.agent.tierName=perf_x_x_processor -Dappdynamics.agent.nodeName=10.45.241.202_perf_x_x_processor_b68124049b2e"Full Agent Registration Info Resolver found system property [appdynamics.agent.applicationName] for application name [it-x-perf]Full Agent Registration Info Resolver found system property [appdynamics.agent.tierName] for tier name [perf_x_x_processor]Full Agent Registration Info Resolver found system property [appdynamics.agent.nodeName] for node name [x_perf_x_x_processor_b68124049b2e]Full Agent Registration Info Resolver using selfService [false]Full Agent Registration Info Resolver using selfService [false]Full Agent Registration Info Resolver using ephemeral node setting [false]Full Agent Registration Info Resolver using application name [it-x-perf]Full Agent Registration Info Resolver using tier name [perf_x_x_processor]Full Agent Registration Info Resolver resolved node name [x_perf_x_x_processor_b68124049b2e]Install Directory resolved to[/opt/app/monitoring/appdynamics]Agent conf directory set to [/opt/app/monitoring/appdynamics/ver20.6.0.30246/conf][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: JavaAgent - Agent conf directory set to [/opt/app/monitoring/appdynamics/ver20.6.0.30246/conf][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver is running[AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found system property [appdynamics.agent.applicationName] for application name [it-x-perf][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found system property [appdynamics.agent.tierName] for tier name [perf_x_x_processor][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver found system property [appdynamics.agent.nodeName] for node name [x_perf_x_x_processor_b68124049b2e][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using selfService [false][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using ephemeral node setting [false][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using application name [it-x-perf][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver using tier name [perf_x_x_processor][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Full Agent Registration Info Resolver resolved node name [x_perf_x_x_processor_b68124049b2e][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[DEBUG]: AgentInstallManager - Full Agent Registration Info Resolver finished running[AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Agent runtime directory set to [/opt/app/monitoring/appdynamics/ver20.6.0.30246][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Agent node directory set to [x_perf_x_x_processor_b68124049b2e]Agent runtime conf directory set to /opt/app/monitoring/appdynamics/ver20.6.0.30246/conf[AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: AgentInstallManager - Agent runtime conf directory set to /opt/app/monitoring/appdynamics/ver20.6.0.30246/conf[AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: JavaAgent - Using Java Agent Version [Server Agent #20.6.0.30246 v20.6.0 GA compatible with 4.4.1.0 r6b0f05b34dd990b8006acb06d3820d0b2cb31f7f master][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: JavaAgent - Running IBM Java Agent [No][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: JavaAgent - Java Agent Directory [/opt/app/monitoring/appdynamics/ver20.6.0.30246][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: JavaAgent - Java Agent AppAgent directory [/opt/app/monitoring/appdynamics/ver20.6.0.30246]Agent logging directory set to [/opt/app/monitoring/appdynamics/ver20.6.0.30246/logs][AD Agent init] Tue Feb 02 18:49:52 UTC 2021[INFO]: JavaAgent - Agent logging directory set to [/opt/app/monitoring/appdynamics/ver20.6.0.30246/logs]Registered app server agent with Node ID[1996663] Component ID[463195] Application ID [8128]Started AppDynamics Java Agent Successfully.18:50:11.926 [main] DEBUG org.springframework.web.client.RestTemplate - HTTP GET x18:50:11.946 [main] DEBUG org.springframework.web.client.RestTemplate - Accept=[text/plain, application/json, application/*+json, */*]18:50:11.962 [main] DEBUG org.springframework.web.client.RestTemplate - Response 200 OK18:50:11.964 [main] DEBUG org.springframework.web.client.RestTemplate - Reading to [java.lang.String] as "text/plain"18:50:11.971 [main] INFO com.x.x.x - Host address is: x18:50:11.973 [main] INFO com.x.x.x - Hostname is: b68124049b2e18:50:11.974 [main] DEBUG org.springframework.web.client.RestTemplate - HTTP GET x18:50:11.975 [main] DEBUG org.springframework.web.client.RestTemplate - Accept=[text/plain, application/json, application/*+json, */*]18:50:11.976 [main] DEBUG org.springframework.web.client.RestTemplate - Response 200 OK18:50:11.976 [main] DEBUG org.springframework.web.client.RestTemplate - Reading to [java.lang.String] as "text/plain;charset=utf-8"18:50:11.978 [main] INFO com.x.x.x - Got task meta: <200,\{ "Arn": "arn:aws:ecs:us-east-1:489420236185:task/x-streamprocessor-perf-main/94b00494b7554f98a286d31b8fea0ca1", "DesiredStatus": "RUNNING", "KnownStatus": "RUNNING", "Family": "x-processor-perf-main", "Version": "119", "Containers": [ { "DockerId": "b68124049b2ec4ef41d07ea3cae1f0cafc554fa2ef965eb1bb392ff74694db35", "DockerName": "ecs-x-processor-perf-main-119-x-streamer-b8fbfa97d5ccc4868f01", "Name": "x-streamer", "Ports": [ { "ContainerPort": 8080, "Protocol": "tcp", "HostPort": 9464 } ], "Volumes": [ \{ "Source": "/var/run/docker.sock", "Destination": "/var/run/docker.sock" }, \{ "Source": "/data/x/perf/main", "Destination": "/data/x/perf/main" } ] } ]},[Date:"Tue, 02 Feb 2021 18:50:11 GMT", Content-Length:"647", Content-Type:"text/plain; charset=utf-8"]>18:50:12.034 [main] INFO com.x.x.x - Host port: 946418:50:12.034 [main] INFO com.x.x.x - Setting instance URL x:9464 ____ __________ ____ ______ __ ____ / __ \/_ __/ __ \ / __ \____ / ____/___ ____/ /__ / __ \_________ ________ ______________ _____ / /_/ / / / / /_/ / / / / / __ \/ / / __ \/ __ / _ \ / /_/ / ___/ __ \/ ___/ _ \/ ___/ ___/ __ \/ ___/ / _, _/ / / / ____(__ |__ ) /_/ / / /_/ |_| /_/ /_/ \____/ .___/\____/\____/\__,_/\___/ /_/ /_/ \____/\___/\___/____/____/\____/_/ /_/ 2021-02-02T18:50:12.960::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.x::Starting x on b68124049b2e with PID 1 (/opt/app/x_x_processor-1.6.19.jar started by root in /opt/app)2021-02-02T18:50:12.961::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.x::The following profiles are active: perf2021-02-02T18:50:15.134::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.b.w.embedded.tomcat.TomcatWebServer::Tomcat initialized with port(s): 8080 (http)2021-02-02T18:50:15.163::AccountId ::Partition ::Offset ::[main]::INFO ::o.apache.coyote.http11.Http11NioProtocol::Initializing ProtocolHandler ["http-nio-0.0.0.0-8080"]2021-02-02T18:50:15.173::AccountId ::Partition ::Offset ::[main]::INFO ::org.apache.catalina.core.StandardService::Starting service [Tomcat]2021-02-02T18:50:15.173::AccountId ::Partition ::Offset ::[main]::INFO ::org.apache.catalina.core.StandardEngine::Starting Servlet engine: [Apache Tomcat/9.0.17]2021-02-02T18:50:15.324::AccountId ::Partition ::Offset ::[main]::INFO ::o.a.c.c.C.[Tomcat].[localhost].[/]::Initializing Spring embedded WebApplicationContext2021-02-02T18:50:15.324::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.web.context.ContextLoader::Root WebApplicationContext: initialization completed in 2305 ms2021-02-02T18:50:16.101::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Took 9 ms to initialize state store 2021-02-02T18:50:16.107::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.serdes.avro.SerializerConfig::Schema registry enabled: true2021-02-02T18:50:16.142::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.KafkaAvroDeserializerConfig::KafkaAvroDeserializerConfig values: bearer.auth.token = [hidden] proxy.port = -1 schema.reflection = false auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL specific.avro.reader = false value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy schema.registry.url = [https://x, https://x] basic.auth.user.info = [hidden] proxy.host = schema.registry.basic.auth.user.info = [hidden] bearer.auth.credentials.source = STATIC_TOKEN key.subject.name.strategy = class io.confluent.kafka.serializers.subject.RecordNameStrategy 2021-02-02T18:50:16.145::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.KafkaAvroDeserializerConfig::KafkaAvroDeserializerConfig values: bearer.auth.token = [hidden] proxy.port = -1 schema.reflection = false auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL specific.avro.reader = false value.subject.name.strategy = class io.confluent.kafka.serializers.subject.RecordNameStrategy schema.registry.url = [https://x, https://x] basic.auth.user.info = [hidden] proxy.host = schema.registry.basic.auth.user.info = [hidden] bearer.auth.credentials.source = STATIC_TOKEN key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy 2021-02-02T18:50:16.168::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.KafkaAvroSerializerConfig::KafkaAvroSerializerConfig values: bearer.auth.token = [hidden] proxy.port = -1 schema.reflection = false auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy schema.registry.url = [https://x, https://x] basic.auth.user.info = [hidden] proxy.host = schema.registry.basic.auth.user.info = [hidden] bearer.auth.credentials.source = STATIC_TOKEN key.subject.name.strategy = class io.confluent.kafka.serializers.subject.RecordNameStrategy 2021-02-02T18:50:16.170::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.KafkaAvroSerializerConfig::KafkaAvroSerializerConfig values: bearer.auth.token = [hidden] proxy.port = -1 schema.reflection = false auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy schema.registry.url = [https://x, https://x] basic.auth.user.info = [hidden] proxy.host = schema.registry.basic.auth.user.info = [hidden] bearer.auth.credentials.source = STATIC_TOKEN key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy 2021-02-02T18:50:16.176::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Topology description Topologies: Sub-topology: 0 Source: SMS_x_EVENTS (topics: [x]) --> TIMESTAMP_APPENDER Processor: TIMESTAMP_APPENDER (stores: []) --> x_PROCESSOR <-- SMS_x_EVENTS Processor: x_PROCESSOR (stores: [STORE_ACCOUNT_STATE]) --> HEADER_APPENDER <-- TIMESTAMP_APPENDER Processor: HEADER_APPENDER (stores: []) --> ACCOUNT_EVENTS <-- x_PROCESSOR Sink: ACCOUNT_EVENTS (topic: ACCOUNT_EVENTS_PERF) <-- HEADER_APPENDER 2021-02-02T18:50:16.179::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Exception handling enabled: true2021-02-02T18:50:16.207::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Exception handling enabled: true2021-02-02T18:50:16.209::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Monitoring enabled: true2021-02-02T18:50:16.210::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Application Server for this instance: x:94642021-02-02T18:50:16.253::AccountId ::Partition ::Offset ::[main]::INFO ::c.s.x.streams.xStreamingJob::Instantiating xStreamingJob2021-02-02T18:50:16.268::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.KafkaAvroDeserializerConfig::KafkaAvroDeserializerConfig values: bearer.auth.token = [hidden] proxy.port = -1 schema.reflection = false auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL specific.avro.reader = false value.subject.name.strategy = class io.confluent.kafka.serializers.subject.RecordNameStrategy schema.registry.url = [https://x, https://x] basic.auth.user.info = [hidden] proxy.host = schema.registry.basic.auth.user.info = [hidden] bearer.auth.credentials.source = STATIC_TOKEN key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy 2021-02-02T18:50:16.280::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.KafkaAvroDeserializerConfig::KafkaAvroDeserializerConfig values: bearer.auth.token = [hidden] proxy.port = -1 schema.reflection = false auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL specific.avro.reader = true value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy schema.registry.url = [https://x, https://x] basic.auth.user.info = [hidden] proxy.host = schema.registry.basic.auth.user.info = [hidden] bearer.auth.credentials.source = STATIC_TOKEN key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy 2021-02-02T18:50:16.288::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Initializing min sequence manager.2021-02-02T18:50:16.290::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Initializing redistributing kafka producer.2021-02-02T18:50:16.601::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.config.ProcessorConfig::Exception handling enabled: true2021-02-02T18:50:17.312::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.b.a.e.web.EndpointLinksResolver::Exposing 2 endpoint(s) beneath base path '/actuator'2021-02-02T18:50:17.403::AccountId ::Partition ::Offset ::[main]::INFO ::i.c.k.s.KafkaAvroSerializerConfig::KafkaAvroSerializerConfig values: bearer.auth.token = [hidden] proxy.port = -1 schema.reflection = false auto.register.schemas = true max.schemas.per.subject = 1000 basic.auth.credentials.source = URL value.subject.name.strategy = class io.confluent.kafka.serializers.subject.RecordNameStrategy schema.registry.url = [https://x, https://x] basic.auth.user.info = [hidden] proxy.host = schema.registry.basic.auth.user.info = [hidden] bearer.auth.credentials.source = STATIC_TOKEN key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy 2021-02-02T18:50:17.426::AccountId ::Partition ::Offset ::[main]::INFO ::s.d.s.w.PropertySourcedRequestMappingHandlerMapping::Mapped URL path [/v2/api-docs] onto method [public org.springframework.http.ResponseEntity<springfox.documentation.spring.web.json.Json> springfox.documentation.swagger2.web.Swagger2Controller.getDocumentation(java.lang.String,javax.servlet.http.HttpServletRequest)]2021-02-02T18:50:17.596::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.concurrent.ThreadPoolTaskExecutor::Initializing ExecutorService 'applicationTaskExecutor'2021-02-02T18:50:18.033::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.web.DefaultSecurityFilterChain::Creating filter chain: Ant [pattern='/v2/api-docs'], []2021-02-02T18:50:18.033::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.web.DefaultSecurityFilterChain::Creating filter chain: Ant [pattern='/configuration/ui'], []2021-02-02T18:50:18.033::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.web.DefaultSecurityFilterChain::Creating filter chain: Ant [pattern='/swagger-resources/**'], []2021-02-02T18:50:18.033::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.web.DefaultSecurityFilterChain::Creating filter chain: Ant [pattern='/configuration/security'], []2021-02-02T18:50:18.033::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.web.DefaultSecurityFilterChain::Creating filter chain: Ant [pattern='/swagger-ui.html'], []2021-02-02T18:50:18.033::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.web.DefaultSecurityFilterChain::Creating filter chain: Ant [pattern='/webjars/**'], []2021-02-02T18:50:18.052::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.s.web.DefaultSecurityFilterChain::Creating filter chain: any request, [org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@aaee2a2, org.springframework.security.web.context.SecurityContextPersistenceFilter@5533dc72, org.springframework.security.web.header.HeaderWriterFilter@5c20aab9, org.springframework.security.web.authentication.logout.LogoutFilter@164a62bf, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@64fc097e, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@2a32fb6, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@f3021cb, org.springframework.security.web.session.SessionManagementFilter@2c768ada, org.springframework.security.web.access.ExceptionTranslationFilter@387bf2d9]2021-02-02T18:50:18.113::AccountId ::Partition ::Offset ::[main]::INFO ::s.d.s.w.p.DocumentationPluginsBootstrapper::Context refreshed2021-02-02T18:50:18.146::AccountId ::Partition ::Offset ::[main]::INFO ::s.d.s.w.p.DocumentationPluginsBootstrapper::Found 1 custom documentation plugin(s)2021-02-02T18:50:18.187::AccountId ::Partition ::Offset ::[main]::INFO ::s.d.s.w.s.ApiListingReferenceScanner::Scanning for api listing references2021-02-02T18:50:18.297::AccountId ::Partition ::Offset ::[main]::INFO ::o.apache.coyote.http11.Http11NioProtocol::Starting ProtocolHandler ["http-nio-0.0.0.0-8080"]2021-02-02T18:50:18.313::AccountId ::Partition ::Offset ::[main]::INFO ::o.s.b.w.embedded.tomcat.TomcatWebServer::Tomcat started on port(s): 8080 (http) with context path ''2021-02-02T18:50:18.316::AccountId ::Partition ::Offset ::[main]::INFO ::com.x.x.x::Started x in 6.203 seconds (JVM running for 26.925)2021-02-02T20:39:09.830::AccountId ::Partition ::Offset ::[com.x.x.streams.xStreamingJob.x-client-StreamThread-2]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread [com.x.x.streams.xStreamingJob.x-client-StreamThread-2] Encountered the following exception during processing and the thread is going to shut down: java.lang.IllegalStateException: Tried to lookup lag for unknown task 0_13 at org.apache.kafka.streams.processor.internals.assignment.ClientState.lagFor(ClientState.java:318) at java.util.Comparator.lambda$comparingLong$6043328a$1(Comparator.java:511) at java.util.Comparator.lambda$thenComparing$36697e65$1(Comparator.java:216) at java.util.TreeMap.put(TreeMap.java:552) at java.util.TreeSet.add(TreeSet.java:255) at java.util.AbstractCollection.addAll(AbstractCollection.java:344) at java.util.TreeSet.addAll(TreeSet.java:312) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.getPreviousTasksByLag(StreamsPartitionAssignor.java:1299) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assignTasksToThreads(StreamsPartitionAssignor.java:1213) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.computeNewAssignment(StreamsPartitionAssignor.java:939) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:393) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:689) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:111) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:599) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:562) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1164) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1139) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1292) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:801) at org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:763) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:614) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)2021-02-02T20:39:09.845::AccountId ::Partition ::Offset ::[com.x.x.streams.xStreamingJob.x-client-StreamThread-2]::INFO ::com.x.streams.xHandler::Got configs as -> \{replication.factor=3, restore.consumer.max.poll.interval.ms=3600000, producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor, commit.interval.ms=30000, topology.optimization=all, confluent.monitoring.interceptor.security.protocol=SSL, APPLICATION=x, bootstrap.servers=x.awscorp.x.com:9092,x.awscorp.x.com:9092,x:9092, metrics.recording.level=DEBUG, default.production.exception.handler=class com.x.streams.IgnoreRecordTooLargeHandler, producer.acks=all, topic.min.insync.replicas=2, confluent.monitoring.interceptor.ssl.truststore.location=/opt/app/client.truststore.jks, enable.auto.commit=true, cache.max.bytes.buffering=10482760, fetch.max.wait.ms=500, DLQ_TOPIC=x_DLQ, application.server=x:9464, ssl.truststore.password=g93eqw, restore.consumer.max.poll.records=1000, session.timeout.ms=1620000, restore.consumer.fetch.max.bytes=10485760, client.id=com.x.x.streams.xStreamingJob.x-client, num.standby.replicas=1, max.poll.records=250, default.deserialization.exception.handler=class com.x.streams.DLQDeserializationExceptionHandler, heartbeat.interval.ms=540000, state.dir=/data/x/perf/main, EXCEPTION_HANDLING_ENABLED=true, security.protocol=SSL, max.request.size=10485760, poll.ms=100, ssl.truststore.location=/opt/app/client.truststore.jks, confluent.monitoring.interceptor.bootstrap.servers=x.awscorp.x.com:9092,x.awscorp.x.com:9092,x:9092, restore.consumer.max.partition.fetch.bytes=10485760, consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor, confluent.monitoring.interceptor.ssl.truststore.password=g93eqw, fetch.min.bytes=1, max.poll.interval.ms=3600000, auto.offset.reset=earliest, num.stream.threads=2, application.id=com.x.x.streams.xStreamingJob.x} Exception: java.lang.NullPointerException thrown from the UncaughtExceptionHandler in thread "com.x.x.streams.xStreamingJob.x-client-StreamThread-2"2021-02-02T21:06:39.903::AccountId ::Partition ::Offset ::[com.x.x.streams.xStreamingJob.x-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread [com.x.x.streams.xStreamingJob.x-client-StreamThread-1] Encountered the following exception during processing and the thread is going to shut down: java.lang.IllegalStateException: Tried to lookup lag for unknown task 0_13 at org.apache.kafka.streams.processor.internals.assignment.ClientState.lagFor(ClientState.java:318) at java.util.Comparator.lambda$comparingLong$6043328a$1(Comparator.java:511) at java.util.Comparator.lambda$thenComparing$36697e65$1(Comparator.java:216) at java.util.TreeMap.put(TreeMap.java:552) at java.util.TreeSet.add(TreeSet.java:255) at java.util.AbstractCollection.addAll(AbstractCollection.java:344) at java.util.TreeSet.addAll(TreeSet.java:312) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.getPreviousTasksByLag(StreamsPartitionAssignor.java:1299) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assignTasksToThreads(StreamsPartitionAssignor.java:1213) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.computeNewAssignment(StreamsPartitionAssignor.java:939) at org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:393) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:689) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:111) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:599) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:562) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1164) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1139) at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1292) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:801) at org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:763) at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:614) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)2021-02-02T21:06:39.915::AccountId ::Partition ::Offset ::[com.x.x.streams.xStreamingJob.x-client-StreamThread-1]::ERROR::org.apache.kafka.streams.KafkaStreams::stream-client [com.x.x.streams.xStreamingJob.x-client] All stream threads have died. The instance will be in error state and should be closed.2021-02-02T21:06:39.915::AccountId ::Partition ::Offset ::[com.x.x.streams.xStreamingJob.x-client-StreamThread-1]::INFO ::com.x.streams.xHandler::Got configs as -> \{replication.factor=3, restore.consumer.max.poll.interval.ms=3600000, producer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor, commit.interval.ms=30000, topology.optimization=all, confluent.monitoring.interceptor.security.protocol=SSL, APPLICATION=x, bootstrap.servers=x.awscorp.x.com:9092,x.awscorp.x.com:9092,x:9092, metrics.recording.level=DEBUG, default.production.exception.handler=class com.x.streams.IgnoreRecordTooLargeHandler, producer.acks=all, topic.min.insync.replicas=2, confluent.monitoring.interceptor.ssl.truststore.location=/opt/app/client.truststore.jks, enable.auto.commit=true, cache.max.bytes.buffering=10482760, fetch.max.wait.ms=500, DLQ_TOPIC=x_DLQ, application.server=x:9464, x, restore.consumer.max.poll.records=1000, session.timeout.ms=1620000, restore.consumer.fetch.max.bytes=10485760, client.id=com.x.x.streams.xStreamingJob.x-client, num.standby.replicas=1, max.poll.records=250, default.deserialization.exception.handler=class com.x.streams.DLQDeserializationExceptionHandler, heartbeat.interval.ms=540000, state.dir=/data/x/perf/main, EXCEPTION_HANDLING_ENABLED=true, security.protocol=SSL, max.request.size=10485760, poll.ms=100, x, confluent.monitoring.interceptor.bootstrap.servers=x.awscorp.x.com:9092,x.awscorp.x.com:9092,x:9092, restore.consumer.max.partition.fetch.bytes=10485760, consumer.interceptor.classes=io.confluent.monitoring.clients.interceptor.MonitoringConsumerInterceptor, confluent.monitoring.interceptor.x, fetch.min.bytes=1, max.poll.interval.ms=3600000, auto.offset.reset=earliest, num.stream.threads=2, application.id=com.x.x.streams.xStreamingJob.x} Exception: java.lang.NullPointerException thrown from the UncaughtExceptionHandler in thread "com.x.x.streams.xStreamingJob.x-client-StreamThread-1" > Kafka 2.7 stream app issue > -------------------------- > > Key: KAFKA-12279 > URL: https://issues.apache.org/jira/browse/KAFKA-12279 > Project: Kafka > Issue Type: Bug > Components: streams > Affects Versions: 2.7.0 > Reporter: prabhu biradar > Priority: Critical > > After starting the stream application below exception is thrown and threads > stop processing. > 2021-02-02T22:29:10.416::AccountId ::Partition ::Offset > ::[com.xx.xx.xx.xx.xx-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread > [com.xx.xx.xx.xx.xx-client-StreamThread-1] Encountered the following > exception during processing and the thread is going to shut down: > 2021-02-02T22:29:10.416::AccountId ::Partition ::Offset > ::[com.xx.xx.xx.xx.xx-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread > [com.xx.xx.xx.xx.xx-client-StreamThread-1] Encountered the following > exception during processing and the thread is going to shut down: > java.lang.IllegalStateException: Tried to lookup lag for unknown task 0_13 at > org.apache.kafka.streams.processor.internals.assignment.ClientState.lagFor(ClientState.java:318) > at java.util.Comparator.lambda$comparingLong$6043328a$1(Comparator.java:511) > at java.util.Comparator.lambda$thenComparing$36697e65$1(Comparator.java:216) > at java.util.TreeMap.put(TreeMap.java:552) at > java.util.TreeSet.add(TreeSet.java:255) at > java.util.AbstractCollection.addAll(AbstractCollection.java:344) at > java.util.TreeSet.addAll(TreeSet.java:312) at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.getPreviousTasksByLag(StreamsPartitionAssignor.java:1299) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assignTasksToThreads(StreamsPartitionAssignor.java:1213) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.computeNewAssignment(StreamsPartitionAssignor.java:939) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:393) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:689) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:111) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:599) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:562) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1164) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1139) > at > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1292) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:801) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:763) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:614) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512)2021-02-02T22:29:10.416::AccountId > ::Partition ::Offset > ::[com.xx.xx.xx.xx.xxx-client-StreamThread-1]::ERROR::o.a.k.s.processor.internals.StreamThread::stream-thread > [com.xx.xx.xx.xx.xxx-client-StreamThread-1] Encountered the following > exception during processing and the thread is going to shut down: > java.lang.IllegalStateException: Tried to lookup lag for unknown task 0_13 at > org.apache.kafka.streams.processor.internals.assignment.ClientState.lagFor(ClientState.java:318) > at java.util.Comparator.lambda$comparingLong$6043328a$1(Comparator.java:511) > at java.util.Comparator.lambda$thenComparing$36697e65$1(Comparator.java:216) > at java.util.TreeMap.put(TreeMap.java:552) at > java.util.TreeSet.add(TreeSet.java:255) at > java.util.AbstractCollection.addAll(AbstractCollection.java:344) at > java.util.TreeSet.addAll(TreeSet.java:312) at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.getPreviousTasksByLag(StreamsPartitionAssignor.java:1299) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assignTasksToThreads(StreamsPartitionAssignor.java:1213) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.computeNewAssignment(StreamsPartitionAssignor.java:939) > at > org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor.assign(StreamsPartitionAssignor.java:393) > at > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:589) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:689) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:111) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:599) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:562) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1164) > at > org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:1139) > at > org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:206) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:169) > at > org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:129) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:602) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:412) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:297) > at > org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) > at > org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1292) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1233) > at > org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:801) > at > org.apache.kafka.streams.processor.internals.StreamThread.pollPhase(StreamThread.java:763) > at > org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:614) > at > org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:553) > at > org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:512) -- This message was sent by Atlassian Jira (v8.3.4#803005)