tanisdlj commented on issue #10868:
URL: https://github.com/apache/druid/issues/10868#issuecomment-776756547
Apparently and from what I can see in the middlemanager and overlord, nope,
they didn't.
The logs from the middlemanager regarding this task:
```
user@druid-dataserver-5$ sudo journalctl -eu druid-middlemanager --since
"2021-02-08 22:28:00" --until "2021-02-08 22:38:00" | grep gnnoiele
Feb 08 22:29:20 druid-dataserver-5 java[25971]: 2021-02-08T22:29:20,129 INFO
[WorkerTaskManager-NoticeHandler]
org.apache.druid.indexing.worker.WorkerTaskManager -
Task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] started.
Feb 08 22:29:20 druid-dataserver-5 java[25971]:
tup.logging.logProperties=true
-Ddruid.indexer.fork.property.druid.lookup.enableLookupSyncOnStartup=true
-Ddruid.zk.service.host=druid-zookeeper-1:2181,druid-zookeeper-2:2181,druid-zookeeper-3:2181
-Ddruid.indexer.logs.bucket=seedtag_analytics_druid_logs
-Ddruid.zk.paths.base=/druid -Dfile.encoding=UTF-8 -Ddruid.storage.type=google
-Ddruid.indexer.fork.property.druid.processing.numMergeBuffers=2
-Ddruid.indexer.logs.type=google -Ddruid.processing.numThreads=2
-Ddruid.processing.buffer.sizeBytes=104857600
-Ddruid.lookup.enableLookupSyncOnStartup=true
-Ddruid.processing.numMergeBuffers=2
-Ddruid.metrics.emitter.dimension.dataSource=ssp-events-hourly
-Ddruid.metrics.emitter.dimension.taskId=index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele
-Ddruid.metrics.emitter.dimension.taskType=index_kafka
-Ddruid.host=druid-dataserver-5 -Ddruid.plaintextPort=8100 -Ddruid.tlsPort=-1
-Ddruid.task.executor.service=druid/middleManager -Ddruid.task
.executor.host=druid-dataserver-5 -Ddruid.task.executor.plaintextPort=8091
-Ddruid.task.executor.enablePlaintextPort=true -Ddruid.task.executor.tlsPort=-1
-Ddruid.task.executor.enableTlsPort=false org.apache.druid.cli.Main internal
peon
/druid/task/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele/task.json
/druid/task/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele/bfdfd9f8-94d1-4c07-8627-455025cd6f6d/status.json
/druid/task/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele/bfdfd9f8-94d1-4c07-8627-455025cd6f6d/report.json
--loadBroadcastSegments true
Feb 08 22:29:20 druid-dataserver-5 java[25971]: 2021-02-08T22:29:20,136 INFO
[forking-task-runner-8] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Logging task index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele output to:
/druid/task/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele/log
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,303 INFO
[qtp40910225-120] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Shutdown [index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] because:
[shut down request via HTTP endpoint]
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,303 INFO
[qtp40910225-120] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Closing output stream to
task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele].
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,332 INFO
[qtp40910225-118] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Shutdown [index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] because:
[shut down request via HTTP endpoint]
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,332 INFO
[qtp40910225-118] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Closing output stream to
task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele].
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,559 INFO
[qtp40910225-132] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Shutdown [index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] because:
[shut down request via HTTP endpoint]
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,559 INFO
[qtp40910225-132] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Closing output stream to
task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele].
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,586 INFO
[qtp40910225-86] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Shutdown [index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] because:
[shut down request via HTTP endpoint]
Feb 08 22:35:36 druid-dataserver-5 java[25971]: 2021-02-08T22:35:36,587 INFO
[qtp40910225-86] org.apache.druid.indexing.overlord.ForkingTaskRunner - Closing
output stream to task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele].
Feb 08 22:36:09 druid-dataserver-5 java[25971]: 2021-02-08T22:36:09,776 INFO
[forking-task-runner-8-[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele]]
org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with
status[2] for task: index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele
Feb 08 22:36:09 druid-dataserver-5 java[25971]: 2021-02-08T22:36:09,778 INFO
[forking-task-runner-8] org.apache.druid.storage.google.GoogleTaskLogs -
Pushing task log
/druid/task/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele/log to:
indexing-logs/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele
Feb 08 22:36:10 druid-dataserver-5 java[25971]: 2021-02-08T22:36:10,019 INFO
[forking-task-runner-8] org.apache.druid.storage.google.GoogleTaskLogs -
Pushing task reports
/druid/task/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele/bfdfd9f8-94d1-4c07-8627-455025cd6f6d/report.json
to:
indexing-logs/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele.report.json
Feb 08 22:36:10 druid-dataserver-5 java[25971]: 2021-02-08T22:36:10,152 INFO
[forking-task-runner-8] org.apache.druid.indexing.overlord.ForkingTaskRunner -
Removing task directory:
/druid/task/index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele
Feb 08 22:36:10 druid-dataserver-5 java[25971]: 2021-02-08T22:36:10,172 INFO
[WorkerTaskManager-NoticeHandler]
org.apache.druid.indexing.worker.WorkerTaskManager - Task
[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] completed with status
[FAILED].
```
From the overlord:
```
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,262 WARN
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor -
All tasks in group [12] failed to publish, killing all tasks for these
partitions
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,262 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown
[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_ddcbmbfg] because: [No task in
the corresponding pending completion taskGroup[12] succeeded before completion
timeout elapsed]
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,277 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to
worker: druid-dataserver-9:8091, status 200 OK, response:
{"task":"index_kafka_ssp-events-hourly_2a3e52186bc4cdd_ddcbmbfg"}
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,277 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.TaskLockbox - Removing
task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_ddcbmbfg] from activeTasks
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,277 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.TaskLockbox - Removing
task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_ddcbmbfg] from
TaskLock[SegmentLock{lockType=EXCLUSIVE,
groupId='index_kafka_ssp-events-hourly', dataSource='ssp-events-hourly',
interval=2021-02-08T20:00:00.000Z/2021-02-08T21:00:00.000Z,
version='2021-02-08T21:14:50.337Z', partitionId=139, priority=75,
revoked=false}]
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,279 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with
id[2032501]: SegmentLock{lockType=EXCLUSIVE,
groupId='index_kafka_ssp-events-hourly', dataSource='ssp-events-hourly',
interval=2021-02-08T20:00:00.000Z/2021-02-08T21:00:00.000Z,
version='2021-02-08T21:14:50.337Z', partitionId=139, priority=75, revoked=false}
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,289 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.MetadataTaskStorage - Updating task
index_kafka_ssp-events-hourly_2a3e52186bc4cdd_ddcbmbfg to status:
TaskStatus{id=index_kafka_ssp-events-hourly_2a3e52186bc4cdd_ddcbmbfg,
status=FAILED, duration=-1, errorMsg=null}
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,297 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.TaskQueue - Task done:
AbstractTask{id='index_kafka_ssp-events-hourly_2a3e52186bc4cdd_ddcbmbfg',
groupId='index_kafka_ssp-events-hourly',
taskResource=TaskResource{availabilityGroup='index_kafka_ssp-events-hourly_2a3e52186bc4cdd',
requiredCapacity=1}, dataSource='ssp-events-hourly',
context={checkpoints={"0":{"12":92380626703}},
IS_INCREMENTAL_HANDOFF_SUPPORTED=true, forceTimeChunkLock=false}}
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,298 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown
[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] because: [No task in
the corresponding pending completion taskGroup[12] succeeded before completion
timeout elapsed]
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,308 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.RemoteTaskRunner - Sent shutdown message to
worker: druid-dataserver-5:8091, status 200 OK, response:
{"task":"index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele"}
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,309 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.TaskLockbox - Removing
task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] from activeTasks
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,309 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.TaskLockbox - Removing
task[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] from
TaskLock[SegmentLock{lockType=EXCLUSIVE,
groupId='index_kafka_ssp-events-hourly', dataSource='ssp-events-hourly',
interval=2021-02-08T20:00:00.000Z/2021-02-08T21:00:00.000Z,
version='2021-02-08T21:14:50.337Z', partitionId=139, priority=75,
revoked=false}]
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,309 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.TaskLockbox - TaskLock is now empty:
SegmentLock{lockType=EXCLUSIVE, groupId='index_kafka_ssp-events-hourly',
dataSource='ssp-events-hourly',
interval=2021-02-08T20:00:00.000Z/2021-02-08T21:00:00.000Z,
version='2021-02-08T21:14:50.337Z', partitionId=139, priority=75, revoked=false}
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,310 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.MetadataTaskStorage - Deleting TaskLock with
id[2032502]: SegmentLock{lockType=EXCLUSIVE,
groupId='index_kafka_ssp-events-hourly', dataSource='ssp-events-hourly',
interval=2021-02-08T20:00:00.000Z/2021-02-08T21:00:00.000Z,
version='2021-02-08T21:14:50.337Z', partitionId=139, priority=75, revoked=false}
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,319 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.MetadataTaskStorage - Updating task
index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele to status:
TaskStatus{id=index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele,
status=FAILED, duration=-1, errorMsg=null}
Feb 8 22:35:36 druid-master-2 java[23035]: 2021-02-08T22:35:36,327 INFO
[NodeRoleWatcher[PEON]]
org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher
- Node[http://druid-dataserver-9:8100] of role[peon] went offline.
```
From the overlord, I would say the important ones are:
```
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,262 WARN
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor -
All tasks in group [12] failed to publish, killing all tasks for these
partitions
Feb 8 22:35:36 druid-master-2 java[15379]: 2021-02-08T22:35:36,298 INFO
[KafkaSupervisor-ssp-events-hourly]
org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown
[index_kafka_ssp-events-hourly_2a3e52186bc4cdd_gnnoiele] because: [No task in
the corresponding pending completion taskGroup[12] succeeded before completion
timeout elapsed]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
