huang2012chao edited a comment on issue #5343: Kafka indexing tasks occasionally fail with 'Stream closed' URL: https://github.com/apache/incubator-druid/issues/5343#issuecomment-413404880 @jihoonson if i insert some data into kafka ,the task will success! just like the task id =[index_kafka_kafka_ins_de42264b8a929b1_cpgcjopj] There many task failed in the KIS,like this : 2018-08-15T12:04:05,611 INFO [qtp1310871029-50] io.druid.indexing.overlord.ForkingTaskRunner - Killing process for task: index_kafka_kafka_ins_a2b40069b9da776_hkjphola 2018-08-15T12:04:05,664 INFO [forking-task-runner-0] io.druid.storage.hdfs.tasklog.HdfsTaskLogs - Writing task log to: hdfs://master:9000/druid/indexing-logs/index_kafka_kafka_ins_a2b40069b9da776_hkjphola 2018-08-15T12:04:07,929 INFO [forking-task-runner-0] io.druid.storage.hdfs.tasklog.HdfsTaskLogs - Wrote task log to: hdfs://master:9000/druid/indexing-logs/index_kafka_kafka_ins_a2b40069b9da776_hkjphola 2018-08-15T12:04:07,929 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Exception caught during execution java.io.IOException: Stream closed at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170) ~[?:1.8.0_181] at java.io.BufferedInputStream.read1(BufferedInputStream.java:291) ~[?:1.8.0_181] at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_181] at java.io.FilterInputStream.read(FilterInputStream.java:107) ~[?:1.8.0_181] at com.google.common.io.ByteStreams.copy(ByteStreams.java:175) ~[guava-16.0.1.jar:?] at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:452) [druid-indexing-service-0.12.1.jar:0.12.1] at io.druid.indexing.overlord.ForkingTaskRunner$1.call(ForkingTaskRunner.java:224) [druid-indexing-service-0.12.1.jar:0.12.1] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] 2018-08-15T12:04:08,279 INFO [forking-task-runner-0] io.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_kafka_kafka_ins_a2b40069b9da776_hkjphola 2018-08-15T12:04:08,783 INFO [WorkerTaskMonitor] io.druid.indexing.worker.WorkerTaskMonitor - Job's finished. Completed [index_kafka_kafka_ins_a2b40069b9da776_hkjphola] with status [FAILED] I guess it happed because there it no message push into kafka? i have used druid 0.10.2.0 ,kafka 2.11-1.0.0,zookeeper 3.4.10. if you find anything error to configure,pelease tell me!  the success log like [index_kafka_kafka_ins_de42264b8a929b1_cpgcjopj]: [index_kafka_kafka_ins_de42264b8a929b1_cpgcjopj.txt](https://github.com/apache/incubator-druid/files/2292232/index_kafka_kafka_ins_de42264b8a929b1_cpgcjopj.txt) in the overlord the failed task log: 2018-08-15T12:04:06,040 INFO [KafkaSupervisor-kafka_ins] io.druid.indexing.overlord.MetadataTaskStorage - Updating task index_kafka_kafka_ins_a2b40069b9da776_hkjphola to status: TaskStatus{id=index_kafka_kafka_ins_a2b40069b9da776_hkjphola, status=FAILED, duration=-1} 2018-08-15T12:04:07,560 WARN [KafkaSupervisor-kafka_disk-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Lag metric: Kafka partitions [0, 1, 2] do not match task partitions [] 2018-08-15T12:04:07,560 INFO [KafkaSupervisor-kafka_disk-Reporting-0] io.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2018-08-15T12:04:07.560Z","service":"druid/overlord","host":"master:8090","version":"0.12.1","metric":"ingest/kafka/lag","value":0,"dataSource":"kafka_disk"}] 2018-08-15T12:04:07,569 WARN [KafkaSupervisor-kafka_ins-Reporting-0] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Lag metric: Kafka partitions [0, 1, 2] do not match task partitions [] 2018-08-15T12:04:07,569 INFO [KafkaSupervisor-kafka_ins-Reporting-0] io.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2018-08-15T12:04:07.569Z","service":"druid/overlord","host":"master:8090","version":"0.12.1","metric":"ingest/kafka/lag","value":0,"dataSource":"kafka_ins"}] 2018-08-15T12:04:08,587 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Worker[slave2:8091] wrote FAILED status for task [index_kafka_kafka_ins_a2b40069b9da776_hkjphola] on [TaskLocation{host='slave2', port=8102, tlsPort=-1}] 2018-08-15T12:04:08,587 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Worker[slave2:8091] completed task[index_kafka_kafka_ins_a2b40069b9da776_hkjphola] with status[FAILED] 2018-08-15T12:04:08,587 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_kafka_kafka_ins_a2b40069b9da776_hkjphola 2018-08-15T12:04:08,954 INFO [KafkaSupervisor-kafka_ins] io.druid.indexing.overlord.TaskQueue - Task done: KafkaIndexTask{id=index_kafka_kafka_ins_a2b40069b9da776_hkjphola, type=index_kafka, dataSource=kafka_ins} 2018-08-15T12:04:08,955 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_kafka_kafka_ins_a2b40069b9da776_hkjphola] on worker[slave2:8091] 2018-08-15T12:04:08,978 WARN [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskQueue - Unknown task completed: index_kafka_kafka_ins_a2b40069b9da776_hkjphola 2018-08-15T12:04:08,978 INFO [Curator-PathChildrenCache-3] io.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2018-08-15T12:04:08.978Z","service":"druid/overlord","host":"master:8090","version":"0.12.1","metric":"task/run/time","value":103077,"dataSource":"kafka_ins","taskStatus":"FAILED","taskType":"index_kafka"}] 2018-08-15T12:04:08,978 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskQueue - Task FAILED: KafkaIndexTask{id=index_kafka_kafka_ins_a2b40069b9da776_hkjphola, type=index_kafka, dataSource=kafka_ins} (103077 run duration) 2018-08-15T12:04:08,978 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_kafka_ins_a2b40069b9da776_hkjphola] status changed to [FAILED]. 2018-08-15T12:04:08,979 INFO [Curator-PathChildrenCache-3] io.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_kafka_ins_a2b40069b9da776_hkjphola] went bye bye. 2018-08-15T12:04:08,990 INFO [KafkaIndexTaskClient-kafka_disk-0] io.druid.indexing.kafka.KafkaIndexTaskClient - No TaskLocation available for task [index_kafka_kafka_disk_8258b8d8c8b7341_fgmoceab], this task may not have been assigned to a worker yet or may have already completed 2018-08-15T12:04:08,992 INFO [KafkaSupervisor-kafka_ins] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id='kafka_ins', generationTime=2018-08-15T12:04:08.992Z, payload={dataSource='kafka_ins', topic='ins', partitions=3, replicas=1, durationSeconds=3600, active=[{id='index_kafka_kafka_ins_a2b40069b9da776_hkjphola', startTime=null, remainingSeconds=null}], publishing=[{id='index_kafka_kafka_ins_de42264b8a929b1_cpgcjopj', startTime=2018-08-15T10:49:33.227Z, remainingSeconds=925}]}} 2018-08-15T12:04:09,001 INFO [KafkaSupervisor-kafka_disk] io.druid.indexing.kafka.supervisor.KafkaSupervisor - {id='kafka_disk', generationTime=2018-08-15T12:04:09.001Z, payload={dataSource='kafka_disk', topic='disk', partitions=3, replicas=1, durationSeconds=3600, active=[{id='index_kafka_kafka_disk_8258b8d8c8b7341_fgmoceab', startTime=null, remainingSeconds=null}], publishing=[{id='index_kafka_kafka_disk_8258b8d8c8b7341_fpojhnlo', startTime=2018-08-15T10:59:25.500Z, remainingSeconds=1650}]}} 2018-08-15T12:04:09,005 WARN [KafkaSupervisor-kafka_ins] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Task [index_kafka_kafka_ins_a2b40069b9da776_hkjphola] failed to return start time, killing task 2018-08-15T12:04:09,317 INFO [KafkaSupervisor-kafka_ins] io.druid.indexing.kafka.supervisor.KafkaSupervisor - Number of tasks [0] does not match configured numReplicas [1] in task group [0], creating more tasks 2018-08-15T12:04:09,318 INFO [KafkaSupervisor-kafka_ins] io.druid.indexing.overlord.MetadataTaskStorage - Inserting task index_kafka_kafka_ins_a2b40069b9da776_epkmnncf with status: TaskStatus{id=index_kafka_kafka_ins_a2b40069b9da776_epkmnncf, status=RUNNING, duration=-1} 2018-08-15T12:04:09,983 INFO [KafkaSupervisor-kafka_ins] io.druid.indexing.overlord.TaskLockbox - Adding task[index_kafka_kafka_ins_a2b40069b9da776_epkmnncf] to activeTasks
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
