Hi Madhu, In HelloWorld app, we add LogHanlder to print log messages to stdout ``addLogHandler(new PrinterLogHandler(new PrintWriter(System.out, true)))``
when we add LogHandler, We start a Kafka Consumer thread in Client that polls Kafka Server (Started when AM Starts) and prints the fetched log messages to stdout. When the AM stops, we stop the kafka server, the stack trace is because there is a race where we make request to fetch message after the Kafka Server has stopped. We can improve the error logging in this case to make this clearer, but this does not impact the functionality. I have created an issue to improve error logging here - https://issues.apache.org/jira/browse/TWILL-151 Thanks Shankar On Mon, Aug 24, 2015 at 1:21 PM, Henry Saputra <[email protected]> wrote: > Hi Madhu, > > This is probably come from embed Kafka running in Twill application. > > Will take a look more to see if I could reproduce. > > - Henry > > On Sun, Aug 23, 2015 at 11:23 PM, Madhu G <[email protected]> wrote: > > I am running the hello world example and running into this error > > > > /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $CP > > org.apache.twill.example.yarn.HelloWorld twill-demo-1.mycluster.com:2181 > > > > > > > > The YARN app runs fine but here is the error > > > > > > 23:11:53.715 [Kafka-Consumer-log-0] INFO > > o.a.t.i.k.client.SimpleKafkaConsumer - Exception when fetching message > on > > TopicPartition{topic=log, partition=0}. > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_67] > > > > at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_67] > > > > at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_67] > > > > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) > > ~[na:1.7.0_67] > > > > at kafka.network.BlockingChannel.connect(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.connect(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.fetch(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.fetchMessages(SimpleKafkaConsumer.java:419) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.run(SimpleKafkaConsumer.java:355) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > > > ---------------------- > > > > To give more context, here is the meaningful part of the logs > > > > > > 2015-08-24T06:11:50,807Z INFO o.a.t.i.a.ApplicationMasterService [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > > ApplicationMasterService:handleCompleted(ApplicationMasterService.java:449) > > - Container container_1440313579459_0019_01_000002 completed with > COMPLETE:. > > > > 2015-08-24T06:11:50,811Z INFO o.a.t.i.a.RunningContainers [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > RunningContainers:handleCompleted(RunningContainers.java:393) - Container > > container_1440313579459_0019_01_000002 exited normally with state > COMPLETE > > > > 2015-08-24T06:11:50,824Z INFO o.a.t.i.a.ApplicationMasterService [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > ApplicationMasterService:doRun(ApplicationMasterService.java:362) - All > > containers completed. Shutting down application master. > > > > 2015-08-24T06:11:50,826Z INFO o.a.t.i.a.ApplicationMasterService [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > ApplicationMasterService:doStop(ApplicationMasterService.java:238) - Stop > > application master with spec: > > > {"name":"HelloWorldRunnable","runnables":{"HelloWorldRunnable":{"name":"HelloWorldRunnable","runnable":{"classname":"org.apache.twill.example.yarn.HelloWorld$HelloWorldRunnable","name":"HelloWorldRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1},"files":[]}},"orders":[{"names":["HelloWorldRunnable"],"type":"STARTED"}],"placementPolicies":[],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}} > > > > 2015-08-24T06:11:50,829Z INFO o.a.t.i.a.RunningContainers [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > RunningContainers:stopAll(RunningContainers.java:332) - Stopping all > > instances of HelloWorldRunnable > > > > 2015-08-24T06:11:50,829Z INFO o.a.t.i.a.RunningContainers [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > RunningContainers:stopAll(RunningContainers.java:342) - Terminated all > > instances of HelloWorldRunnable > > > > 2015-08-24T06:11:50,848Z INFO o.a.t.i.a.ApplicationMasterService [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > ApplicationMasterService:cleanupDir(ApplicationMasterService.java:326) - > > Application directory deleted: hdfs:// > > > twill-demo-1.mycluster.com:8020/user/root/HelloWorldRunnable/526a193b-e9e0-40ce-99f9-a79b174d2871 > > > > 2015-08-24T06:11:50,848Z INFO o.a.t.i.AbstractTwillService [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > AbstractTwillService:removeLiveNode(AbstractTwillService.java:209) - > Remove > > live node > > > twill-demo-1.mycluster.com:2181/HelloWorldRunnable/instances/526a193b-e9e0-40ce-99f9-a79b174d2871 > > > > 2015-08-24T06:11:50,851Z INFO o.a.t.i.AbstractTwillService [ > > twill-demo-2.mycluster.com] [ApplicationMasterService] > > AbstractTwillService:shutDown(AbstractTwillService.java:190) - Service > > ApplicationMasterService with runId 526a193b-e9e0-40ce-99f9-a79b174d2871 > > shutdown completed > > > > 2015-08-24T06:11:50,852Z INFO o.a.t.i.ServiceMain [ > > twill-demo-2.mycluster.com] [main] > ServiceMain:doMain(ServiceMain.java:100) > > - Service ApplicationMasterService [TERMINATED] completed. > > > > 23:11:51.859 [IPC Parameter Sending Thread #0] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root sending #48 > > > > 23:11:51.861 [IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032from root] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root got value #48 > > > > 23:11:51.861 > > [HelloWorldRunnable-application_1440313579459_0019-yarn-poller] DEBUG > > o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 2ms > > > > 23:11:51.863 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Reading reply > > sessionid:0x14eb5f8db18012f, packet:: > > > clientPath:/HelloWorldRunnable/instances/526a193b-e9e0-40ce-99f9-a79b174d2871 > > > serverPath:/HelloWorldRunnable/instances/526a193b-e9e0-40ce-99f9-a79b174d2871 > > finished:false header:: 34,3 replyHeader:: 34,139514,-101 request:: > > '/HelloWorldRunnable/instances/526a193b-e9e0-40ce-99f9-a79b174d2871,F > > response:: > > > > 23:11:52.864 [IPC Parameter Sending Thread #0] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root sending #49 > > > > 23:11:52.865 [IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032from root] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root got value #49 > > > > 23:11:52.866 > > [HelloWorldRunnable-application_1440313579459_0019-yarn-poller] DEBUG > > o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 3ms > > > > 23:11:52.866 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Got notification > > sessionid:0x14eb5f8db18012f > > > > 23:11:52.866 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected > > type:NodeDeleted > > > path:/HelloWorldRunnable/526a193b-e9e0-40ce-99f9-a79b174d2871/kafka/brokers/ids/1 > > for sessionid 0x14eb5f8db18012f > > > > 23:11:52.868 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Reading reply > > sessionid:0x14eb5f8db18012f, packet:: > > > clientPath:/HelloWorldRunnable/instances/526a193b-e9e0-40ce-99f9-a79b174d2871 > > > serverPath:/HelloWorldRunnable/instances/526a193b-e9e0-40ce-99f9-a79b174d2871 > > finished:false header:: 35,3 replyHeader:: 35,139515,-101 request:: > > '/HelloWorldRunnable/instances/526a193b-e9e0-40ce-99f9-a79b174d2871,F > > response:: > > > > 23:11:53.180 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Got notification > > sessionid:0x14eb5f8db18012f > > > > 23:11:53.180 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Got WatchedEvent state:SyncConnected > > type:NodeDeleted > > > path:/HelloWorldRunnable/526a193b-e9e0-40ce-99f9-a79b174d2871/kafka/brokers/topics/log/partitions/0/state > > for sessionid 0x14eb5f8db18012f > > > > 15/08/23 23:11:53 INFO consumer.SimpleConsumer: Reconnect due to socket > > error: Connection reset by peer > > > > 23:11:53.715 [Kafka-Consumer-log-0] INFO > > o.a.t.i.k.client.SimpleKafkaConsumer - Exception when fetching message > on > > TopicPartition{topic=log, partition=0}. > > > > java.net.ConnectException: Connection refused > > > > at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_67] > > > > at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_67] > > > > at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_67] > > > > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) > > ~[na:1.7.0_67] > > > > at kafka.network.BlockingChannel.connect(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.connect(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.reconnect(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.liftedTree1$1(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(Unknown > > Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.metrics.KafkaTimer.time(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.consumer.SimpleConsumer.fetch(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at kafka.javaapi.consumer.SimpleConsumer.fetch(Unknown Source) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.fetchMessages(SimpleKafkaConsumer.java:419) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > at > > > org.apache.twill.internal.kafka.client.SimpleKafkaConsumer$ConsumerThread.run(SimpleKafkaConsumer.java:355) > > > ~[twill-examples-yarn-0.7.0-incubating-SNAPSHOT.jar:0.7.0-incubating-SNAPSHOT] > > > > 23:11:53.719 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Reading reply > > sessionid:0x14eb5f8db18012f, packet:: > > > clientPath:/HelloWorldRunnable/526a193b-e9e0-40ce-99f9-a79b174d2871/kafka/brokers/ids/1 > > > serverPath:/HelloWorldRunnable/526a193b-e9e0-40ce-99f9-a79b174d2871/kafka/brokers/ids/1 > > finished:false header:: 36,3 replyHeader:: 36,139540,-101 request:: > > > '/HelloWorldRunnable/526a193b-e9e0-40ce-99f9-a79b174d2871/kafka/brokers/ids/1,T > > response:: > > > > 23:11:53.720 [Kafka-Consumer-log-0] DEBUG > > o.a.t.i.k.client.SimpleKafkaConsumer - No leader for topic partition > > TopicPartition{topic=log, partition=0}. > > > > 23:11:53.870 [IPC Parameter Sending Thread #0] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root sending #50 > > > > 23:11:53.872 [IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032from root] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root got value #50 > > > > 23:11:53.872 > > [HelloWorldRunnable-application_1440313579459_0019-yarn-poller] DEBUG > > o.a.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 2ms > > > > 23:11:53.872 > > [HelloWorldRunnable-application_1440313579459_0019-yarn-poller] DEBUG > > o.a.twill.yarn.YarnTwillController - Stop polling status from Yarn for > > HelloWorldRunnable application_1440313579459_0019. > > > > 23:11:53.873 > > [HelloWorldRunnable-application_1440313579459_0019-yarn-poller] INFO > > o.a.twill.yarn.YarnTwillController - Yarn application HelloWorldRunnable > > application_1440313579459_0019 completed. Shutting down controller. > > > > 23:11:53.879 [IPC Parameter Sending Thread #0] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root sending #51 > > > > 23:11:53.880 [IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032from root] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root got value #51 > > > > 23:11:53.880 [ STOPPING] DEBUG o.a.hadoop.ipc.ProtobufRpcEngine - Call: > > getApplicationReport took 1ms > > > > 23:11:53.881 [ STOPPING] DEBUG o.a.twill.yarn.YarnTwillController - Yarn > > application HelloWorldRunnable application_1440313579459_0019 completed > > with status SUCCEEDED > > > > 23:11:53.881 [ STOPPING] INFO o.a.t.i.k.client.SimpleKafkaConsumer - > > Requesting stop of all consumer threads. > > > > 23:11:53.881 [ STOPPING] INFO o.a.t.i.k.client.SimpleKafkaConsumer - > > Terminate requested Kafka-Consumer-log-0 > > > > 23:11:53.882 [ STOPPING] INFO o.a.t.i.k.client.SimpleKafkaConsumer - > Wait > > for all consumer threads to stop. > > > > 23:11:53.882 [ STOPPING] INFO o.a.t.i.k.client.SimpleKafkaConsumer - All > > consumer threads stopped. > > > > 23:11:53.886 [ZKKafkaClientService STOPPING] INFO > > o.a.t.i.k.c.ZKKafkaClientService - Stopping KafkaClientService > > > > 23:11:53.886 [ZKKafkaClientService STOPPING] INFO > > o.a.t.i.k.client.SimpleKafkaConsumer - Stopping Kafka consumer > > > > 23:11:53.886 [ZKKafkaClientService STOPPING] INFO > > o.a.t.i.k.client.SimpleKafkaConsumer - Kafka Consumer stopped > > > > 23:11:53.888 [ZKKafkaClientService STOPPING] INFO > > o.a.t.i.k.c.ZKKafkaClientService - KafkaClientService stopped > > > > 23:11:53.960 [Thread-2] DEBUG org.apache.hadoop.ipc.Client - stopping > > client from cache: org.apache.hadoop.ipc.Client@231556b > > > > 23:11:53.975 [zk-client-EventThread] DEBUG > org.apache.zookeeper.ZooKeeper - > > Closing session: 0x14eb5f8db18012f > > > > 23:11:53.975 [zk-client-EventThread] DEBUG > org.apache.zookeeper.ClientCnxn > > - Closing client for session: 0x14eb5f8db18012f > > > > 23:11:53.977 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - Reading reply > > sessionid:0x14eb5f8db18012f, packet:: clientPath:null serverPath:null > > finished:false header:: 37,-11 replyHeader:: 37,139541,0 request:: null > > response:: null > > > > 23:11:53.977 [zk-client-EventThread] DEBUG > org.apache.zookeeper.ClientCnxn > > - Disconnecting client for session: 0x14eb5f8db18012f > > > > 23:11:53.977 [zk-client-EventThread] INFO > org.apache.zookeeper.ZooKeeper - > > Session: 0x14eb5f8db18012f closed > > > > 23:11:53.977 [ STARTING-SendThread(twill-demo-1.mycluster.com:2181)] > DEBUG > > org.apache.zookeeper.ClientCnxn - An exception was thrown while closing > > send thread for session 0x14eb5f8db18012f : Unable to read additional > data > > from server sessionid 0x14eb5f8db18012f, likely server has closed socket > > > > 23:11:53.977 [ STARTING-EventThread] INFO > org.apache.zookeeper.ClientCnxn > > - EventThread shut down > > > > 23:11:53.978 [Hadoop21YarnAppClient STOPPING] DEBUG > > o.a.hadoop.service.AbstractService - Service: > > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state > STOPPED > > > > 23:11:53.978 [Hadoop21YarnAppClient STOPPING] DEBUG > > org.apache.hadoop.ipc.Client - stopping client from cache: > > org.apache.hadoop.ipc.Client@231556b > > > > 23:11:53.978 [Hadoop21YarnAppClient STOPPING] DEBUG > > org.apache.hadoop.ipc.Client - removing client from cache: > > org.apache.hadoop.ipc.Client@231556b > > > > 23:11:53.978 [Hadoop21YarnAppClient STOPPING] DEBUG > > org.apache.hadoop.ipc.Client - stopping actual client because no more > > references remain: org.apache.hadoop.ipc.Client@231556b > > > > 23:11:53.978 [Hadoop21YarnAppClient STOPPING] DEBUG > > org.apache.hadoop.ipc.Client - Stopping client > > > > 23:11:53.979 [IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032from root] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root: closed > > > > 23:11:53.979 [IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032from root] DEBUG > > org.apache.hadoop.ipc.Client - IPC Client (2014438912) connection to > > twill-demo-1.mycluster.com/172.26.22.109:8032 from root: stopped, > remaining > > connections 0 >
