scottbelden opened a new issue #8827: Failure to ingest local file when running through the tutorial with the docker cluster URL: https://github.com/apache/incubator-druid/issues/8827 ### Affected Version 0.16.0-incubating ### Description I am using the docker-compose file located [here](https://github.com/apache/incubator-druid/blob/master/distribution/docker/docker-compose.yml) to start a druid cluster. I modified the compose file to specify the `0.16.0-incubating` tag for the druid images. I am trying to follow the tutorial to [ingest a local file](https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) but after submitting the job it never succeeds and instead always fails. Below are the docker-compose logs for the cluster from the time I submitted the task to the time that it failed: ```overlord | 2019-11-05T22:54:22,032 INFO [qtp2107873140-89] org.apache.druid.indexing.overlord.MetadataTaskStorage - Inserting task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z with status: TaskStatus{id=index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z, status=RUNNING, duration=-1, errorMsg=null} overlord | 2019-11-05T22:54:22,055 INFO [qtp2107873140-89] org.apache.druid.indexing.overlord.TaskLockbox - Adding task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] to activeTasks overlord | 2019-11-05T22:54:22,055 INFO [TaskQueue-Manager] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - [forceTimeChunkLock] is set to true in task context. Use timeChunk lock overlord | 2019-11-05T22:54:22,056 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z overlord | 2019-11-05T22:54:22,056 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Added pending task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z overlord | 2019-11-05T22:54:22,058 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.RemoteTaskRunner - Coordinator asking Worker[172.19.0.6:8091] to add task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] overlord | 2019-11-05T22:54:22,061 DEBUG [qtp2107873140-89] org.apache.druid.jetty.RequestLog - 172.19.0.7 POST //172.19.0.4:8081/druid/indexer/v1/task HTTP/1.1 router | 2019-11-05T22:54:22,063 DEBUG [AsyncManagementForwardingServlet-2b1cd7bc-177] org.apache.druid.jetty.RequestLog - 172.19.0.1 POST //localhost:4008/druid/indexer/v1/task HTTP/1.1 overlord | 2019-11-05T22:54:22,076 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z switched from pending to running (on [172.19.0.6:8091]) overlord | 2019-11-05T22:54:22,077 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [RUNNING]. middlemanager | 2019-11-05T22:54:22,092 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] started. middlemanager | 2019-11-05T22:54:22,095 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Running command: java -cp /tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/jackson-databind-2.6.7.jar:lib/jackson-mapper-asl-1.9.13.jar:lib/commons-lang-2.6.jar:lib/google-oauth-client-1.22.0.jar:lib/jcl-over-slf4j-1.7.12.jar:lib/log4j-1.2-api-2.5.jar:lib/rhino-1.7.11.jar:lib/maven-model-builder-3.1.1.jar:lib/joda-time-2.10.2.jar:lib/druid-console-0.16.0-incubating.jar:lib/zookeeper-3.4.14.jar:lib/maven-repository-metadata-3.1.1.jar:lib/avatica-server-1.12.0.jar:lib/jsr311-api-1.1.1.jar:lib/jackson-jaxrs-json-provider-2.6.7.jar:lib/jetty-servlet-9.4.10.v20180503.jar:lib/druid-sql-0.16.0-incubating.jar:lib/druid-indexing-hadoop-0.16.0-incubating.jar:lib/checker-qual-2.5.7.jar:lib/reactive-streams-1.0.2.jar:lib/derbynet-10.14.2.0.jar:lib/protobuf-java-3.1.0.jar:lib/istack-commons-runtime-3.0.7.jar:lib/json-path-2.3.0.jar:lib/netty-3.10.6.Final.jar:lib/esri-geometry-api-2.0.0.jar:lib/jackson-datatype-guava-2.6.7.jar:lib/aws-java-sdk-kms-1.11.199.jar:lib/asm-analysis-7.1.jar:lib/jboss-logging-3.1.3.GA.jar:lib/hibernate-validator-5.1.3.Final.jar:lib/jackson-module-guice-2.6.7.jar:lib/druid-services-0.16.0-incubating.jar:lib/guice-4.1.0.jar:lib/jackson-annotations-2.6.7.jar:lib/netty-handler-4.1.30.Final.jar:lib/netty-codec-http-4.1.30.Final.jar:lib/commons-math3-3.6.1.jar:lib/jackson-jq-0.0.10.jar:lib/guice-multibindings-4.1.0.jar:lib/netty-buffer-4.1.30.Final.jar:lib/spymemcached-2.12.3.jar:lib/javax.inject-1.jar:lib/jetty-security-9.4.10.v20180503.jar:lib/aws-java-sdk-ec2-1.11.199.jar:lib/config-magic-0.9.jar:lib/maven-settings-builder-3.1.1.jar:lib/wagon-provider-api-2.4.jar:lib/curator-client-4.1.0.jar:lib/jackson-dataformat-cbor-2.6.7.jar:lib/curator-recipes-4.1.0.jar:lib/guice-servlet-4.1.0.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/audience-annotations-0.5.0.jar:lib/avatica-metrics-1.12.0.jar:lib/netty-handler-proxy-4.1.30.Final.jar:lib/joni-2.1.27.jar:lib/xz-1.8.jar:lib/metrics-core-4.0.0.jar:lib/ion-java-1.0.2.jar:lib/javax.el-api-3.0.0.jar:lib/jcodings-1.0.43.jar:lib/jetty-client-9.4.10.v20180503.jar:lib/commons-dbcp2-2.0.1.jar:lib/curator-framework-4.1.0.jar:lib/commons-text-1.3.jar:lib/netty-reactive-streams-2.0.0.jar:lib/stax-ex-1.8.jar:lib/druid-server-0.16.0-incubating.jar:lib/netty-codec-4.1.30.Final.jar:lib/druid-gcp-common-0.16.0-incubating.jar:lib/commons-compress-1.18.jar:lib/janino-2.7.6.jar:lib/jetty-util-9.4.10.v20180503.jar:lib/slf4j-api-1.7.25.jar:lib/netty-transport-native-epoll-4.1.30.Final-linux-x86_64.jar:lib/jsr305-2.0.1.jar:lib/icu4j-54.1.1.jar:lib/accessors-smart-1.2.jar:lib/error_prone_annotations-2.3.2.jar:lib/commons-logging-1.1.1.jar:lib/jetty-continuation-9.4.10.v20180503.jar:lib/jersey-server-1.19.3.jar:lib/shims-0.8.6.jar:lib/commons-net-3.6.jar:lib/httpcore-4.4.11.jar:lib/commons-io-2.6.jar:lib/javax.activation-api-1.2.0.jar:lib/netty-common-4.1.30.Final.jar:lib/okhttp-1.0.2.jar:lib/fastutil-8.2.3.jar:lib/jersey-guice-1.19.3.jar:lib/jersey-servlet-1.19.3.jar:lib/jvm-attach-api-1.5.jar:lib/jackson-datatype-joda-2.6.7.jar:lib/jline-0.9.94.jar:lib/FastInfoset-1.2.15.jar:lib/aws-java-sdk-s3-1.11.199.jar:lib/sigar-1.6.5.132.jar:lib/netty-transport-4.1.30.Final.jar:lib/jackson-dataformat-smile-2.6.7.jar:lib/commons-codec-1.12.jar:lib/netty-resolver-dns-4.1.30.Final.jar:lib/opencsv-4.6.jar:lib/log4j-slf4j-impl-2.5.jar:lib/jackson-core-asl-1.9.13.jar:lib/derby-10.14.2.0.jar:lib/druid-hll-0.16.0-incubating.jar:lib/guava-16.0.1.jar:lib/jdbi-2.63.1.jar:lib/antlr4-runtime-4.5.1.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/RoaringBitmap-0.8.6.jar:lib/validation-api-1.1.0.Final.jar:lib/disruptor-3.3.6.jar:lib/json-smart-2.3.jar:lib/netty-transport-native-unix-common-4.1.30.Final.jar:lib/aopalliance-1.0.jar:lib/jackson-module-jaxb-annotations-2.6.7.jar:lib/google-api-client-1.22.0.jar:lib/calcite-core-1.17.0.jar:lib/lz4-java-1.6.0.jar:lib/jmespath-java-1.11.199.jar:lib/aether-util-0.9.0.M2.jar:lib/druid-aws-common-0.16.0-incubating.jar:lib/derbyclient-10.14.2.0.jar:lib/jackson-jaxrs-base-2.6.7.jar:lib/commons-lang3-3.8.1.jar:lib/jetty-http-9.4.10.v20180503.jar:lib/calcite-linq4j-1.17.0.jar:lib/commons-collections4-4.2.jar:lib/jna-4.5.1.jar:lib/compress-lzf-1.0.4.jar:lib/jaxb-api-2.3.1.jar:lib/log4j-core-2.5.jar:lib/tesla-aether-0.0.5.jar:lib/aggdesigner-algorithm-6.0.jar:lib/jetty-io-9.4.10.v20180503.jar:lib/aws-java-sdk-core-1.11.199.jar:lib/druid-processing-0.16.0-incubating.jar:lib/javax.activation-1.2.0.jar:lib/commons-collections-3.2.2.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/jackson-jaxrs-smile-provider-2.6.7.jar:lib/maven-aether-provider-3.1.1.jar:lib/asm-7.1.jar:lib/txw2-2.3.1.jar:lib/maven-settings-3.1.1.jar:lib/asm-commons-7.1.jar:lib/jetty-servlets-9.4.10.v20180503.jar:lib/asm-tree-7.1.jar:lib/aether-spi-0.9.0.M2.jar:lib/plexus-utils-3.0.15.jar:lib/caffeine-2.5.5.jar:lib/netty-codec-dns-4.1.30.Final.jar:lib/netty-codec-socks-4.1.30.Final.jar:lib/log4j-jul-2.5.jar:lib/httpclient-4.5.3.jar:lib/jaxb-runtime-2.3.1.jar:lib/async-http-client-2.5.3.jar:lib/airline-0.7.jar:lib/classmate-1.0.0.jar:lib/zstd-jni-1.3.3-1.jar:lib/extendedset-0.16.0-incubating.jar:lib/commons-pool2-2.2.jar:lib/druid-core-0.16.0-incubating.jar:lib/log4j-api-2.5.jar:lib/javax.el-3.0.0.jar:lib/aether-impl-0.9.0.M2.jar:lib/jetty-server-9.4.10.v20180503.jar:lib/javax.servlet-api-3.1.0.jar:lib/curator-x-discovery-4.1.0.jar:lib/jackson-core-2.6.7.jar:lib/jersey-core-1.19.3.jar:lib/avatica-core-1.12.0.jar:lib/commons-beanutils-1.9.3.jar:lib/netty-resolver-4.1.30.Final.jar:lib/google-http-client-jackson2-1.22.0.jar:lib/maven-artifact-3.6.0.jar:lib/commons-compiler-2.7.6.jar:lib/maven-model-3.1.1.jar:lib/plexus-interpolation-1.19.jar:lib/druid-indexing-service-0.16.0-incubating.jar:lib/google-http-client-1.22.0.jar:lib/aether-api-0.9.0.M2.jar:lib/jetty-proxy-9.4.10.v20180503.jar: -server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -server -Xmx1g -Xms1g -XX:MaxDirectMemorySize=3g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Ddruid.indexer.task.baseTaskDir=var/druid/task -Ddruid.host=172.19.0.6 -Ddruid.metadata.storage.host= -Ddruid.metadata.storage.connector.password=FoolishPassword -Ddruid.metadata.storage.connector.host=localhost -Ddruid.indexer.fork.property.druid.processing.numThreads=1 -Ddruid.emitter.logging.logLevel=debug -Ddruid.emitter=noop -Ddruid.azure.key=YOURKEY -Ddruid.indexer.fork.property.druid.processing.buffer.sizeBytes=268435456 -Duser.timezone=UTC -Dfile.encoding.pkg=sun.io -Ddruid.azure.maxTries=3 -Ddruid.storage.storageDirectory=var/druid/segments -Ddruid.selectors.coordinator.serviceName=druid/coordinator -Ddruid.selectors.indexing.serviceName=druid/overlord -Ddruid.indexing.doubleStorage=double -Ddruid.lookup.enableLookupSyncOnStartup=false -Ddruid.server.http.numThreads=60 -Ddruid.worker.capacity=4 -Ddruid.metadata.storage.connector.port=1527 -Ddruid.azure.container=druid -Ddruid.azure.account=YOURACCOUNT -Ddruid.service=druid/middleManager -Ddruid.metadata.storage.connector.user=druid -Ddruid.metadata.storage.type=postgresql -Ddruid.metadata.storage.connector.connectURI=jdbc:postgresql://postgres:5432/druid -Ddruid.coordinator.balancer.strategy=cachingCost -Ddruid.plaintextPort=8091 -Djava.io.tmpdir=var/tmp -Ddruid.azure.protocol=https -Ddruid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "druid-azure-extensions", "postgresql-metadata-storage"] -Ddruid.sql.enable=true -Ddruid.startup.logging.logProperties=true -Ddruid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"] -Ddruid.zk.service.host=zookeeper -Ddruid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"] -Ddruid.indexer.logs.directory=var/druid/indexing-logs -Ddruid.zk.paths.base=/druid -Dfile.encoding=UTF-8 -Ddruid.storage.type=azure -Ddruid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp -Ddruid.indexer.fork.property.druid.processing.numMergeBuffers=2 -Ddruid.indexer.logs.type=file -Ddruid.processing.numThreads=1 -Ddruid.processing.buffer.sizeBytes=268435456 -Ddruid.processing.numMergeBuffers=2 -Ddruid.metrics.emitter.dimension.dataSource=wikiticker-2015-09-12-sampled -Ddruid.metrics.emitter.dimension.taskId=index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z -Ddruid.metrics.emitter.dimension.taskType=index_parallel -Ddruid.host=172.19.0.6 -Ddruid.plaintextPort=8100 -Ddruid.tlsPort=-1 -Ddruid.task.executor.service=druid/middleManager -Ddruid.task.executor.host=172.19.0.6 -Ddruid.task.executor.plaintextPort=8091 -Ddruid.task.executor.enablePlaintextPort=true -Ddruid.task.executor.tlsPort=-1 -Ddruid.task.executor.enableTlsPort=false org.apache.druid.cli.Main internal peon var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/task.json var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/e509c961-9133-4e7b-9ed3-a44beb8b5d7b/status.json var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/e509c961-9133-4e7b-9ed3-a44beb8b5d7b/report.json overlord | 2019-11-05T22:54:22,106 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] wrote RUNNING status for task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on [TaskLocation{host='null', port=-1, tlsPort=-1}] middlemanager | 2019-11-05T22:54:22,122 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] location changed to [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}]. middlemanager | 2019-11-05T22:54:22,122 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [RUNNING]. middlemanager | 2019-11-05T22:54:22,122 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Logging task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z output to: var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z/log overlord | 2019-11-05T22:54:22,132 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] wrote RUNNING status for task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}] overlord | 2019-11-05T22:54:22,132 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] location changed to [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}]. coordinator | 2019-11-05T22:54:22,486 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:22,487 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. overlord | 2019-11-05T22:54:23,391 DEBUG [qtp2107873140-64] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/supervisor?full HTTP/1.1 router | 2019-11-05T22:54:23,395 DEBUG [qtp883862737-137] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/supervisor?full HTTP/1.1 overlord | 2019-11-05T22:54:23,407 DEBUG [qtp2107873140-78] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/waitingTasks HTTP/1.1 overlord | 2019-11-05T22:54:23,407 DEBUG [qtp2107873140-86] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/runningTasks HTTP/1.1 router | 2019-11-05T22:54:23,410 DEBUG [qtp883862737-143] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/waitingTasks HTTP/1.1 router | 2019-11-05T22:54:23,411 DEBUG [qtp883862737-142] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/runningTasks HTTP/1.1 router | 2019-11-05T22:54:23,418 DEBUG [AsyncManagementForwardingServlet-2b1cd7bc-173] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/completeTasks HTTP/1.1 overlord | 2019-11-05T22:54:23,419 DEBUG [qtp2107873140-72] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/completeTasks HTTP/1.1 overlord | 2019-11-05T22:54:23,420 DEBUG [qtp2107873140-61] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.4:8081/druid/indexer/v1/pendingTasks HTTP/1.1 router | 2019-11-05T22:54:23,423 DEBUG [qtp883862737-99] org.apache.druid.jetty.RequestLog - 172.19.0.1 GET //localhost:4008/druid/indexer/v1/pendingTasks HTTP/1.1 coordinator | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:27,488 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. coordinator | 2019-11-05T22:54:32,489 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:32,490 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:32,491 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. overlord | 2019-11-05T22:54:36,182 INFO [TaskQueue-StorageSync] org.apache.druid.indexing.overlord.TaskQueue - Synced 1 tasks from storage (0 tasks added, 0 tasks removed). coordinator | 2019-11-05T22:54:37,493 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:37,493 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:37,493 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:37,494 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:37,494 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:37,494 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. coordinator | 2019-11-05T22:54:39,017 INFO [DatabaseRuleManager-Exec--0] org.apache.druid.metadata.SQLMetadataRuleManager - Polled and found 1 rule(s) for 1 datasource(s) coordinator | 2019-11-05T22:54:40,178 INFO [org.apache.druid.metadata.SQLMetadataSegmentManager-Exec--0] org.apache.druid.metadata.SQLMetadataSegmentManager - No segments found in the database! coordinator | 2019-11-05T22:54:42,496 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:42,496 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:42,497 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. coordinator | 2019-11-05T22:54:44,828 DEBUG [qtp1733520096-108] org.apache.druid.jetty.RequestLog - 172.19.0.7 GET //172.19.0.5:8081/druid/coordinator/v1/rules HTTP/1.1 coordinator | 2019-11-05T22:54:47,498 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:47,498 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:47,499 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. coordinator | 2019-11-05T22:54:52,500 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:52,500 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:52,501 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:52,501 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:52,501 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:52,502 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. middlemanager | 2019-11-05T22:54:57,128 INFO [forking-task-runner-1-[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z]] org.apache.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[1] for task: index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z middlemanager | 2019-11-05T22:54:57,128 INFO [forking-task-runner-1] org.apache.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: var/druid/indexing-logs/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z.log middlemanager | 2019-11-05T22:54:57,129 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [FAILED]. middlemanager | 2019-11-05T22:54:57,129 INFO [forking-task-runner-1] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z middlemanager | 2019-11-05T22:54:57,144 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Job's finished. Completed [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] with status [FAILED] overlord | 2019-11-05T22:54:57,146 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] wrote FAILED status for task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on [TaskLocation{host='172.19.0.6', port=8100, tlsPort=-1}] overlord | 2019-11-05T22:54:57,147 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[172.19.0.6:8091] completed task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] with status[FAILED] overlord | 2019-11-05T22:54:57,148 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z overlord | 2019-11-05T22:54:57,149 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] because: [notified status change from task] overlord | 2019-11-05T22:54:57,149 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] on worker[172.19.0.6:8091] overlord | 2019-11-05T22:54:57,153 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] from activeTasks overlord | 2019-11-05T22:54:57,154 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.MetadataTaskStorage - Updating task index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z to status: TaskStatus{id=index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z, status=FAILED, duration=35039, errorMsg=null} overlord | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task done: AbstractTask{id='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', groupId='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', taskResource=TaskResource{availabilityGroup='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', requiredCapacity=1}, dataSource='wikiticker-2015-09-12-sampled', context={forceTimeChunkLock=true}} overlord | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task FAILED: AbstractTask{id='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', groupId='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', taskResource=TaskResource{availabilityGroup='index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z', requiredCapacity=1}, dataSource='wikiticker-2015-09-12-sampled', context={forceTimeChunkLock=true}} (35039 run duration) overlord | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] status changed to [FAILED]. overlord | 2019-11-05T22:54:57,160 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task[index_parallel_wikiticker-2015-09-12-sampled_2019-11-05T22:54:22.026Z] went bye bye. coordinator | 2019-11-05T22:54:57,503 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:54:57,503 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:54:57,504 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:54:57,504 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:54:57,504 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:54:57,505 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served. coordinator | 2019-11-05T22:55:02,506 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Starting coordination. Getting used segments. coordinator | 2019-11-05T22:55:02,506 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorSegmentInfoLoader - Found [0] used segments. coordinator | 2019-11-05T22:55:02,507 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty. coordinator | 2019-11-05T22:55:02,507 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorBalancer - Metadata segments are not available. Cannot balance. coordinator | 2019-11-05T22:55:02,508 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Load Queues: coordinator | 2019-11-05T22:55:02,508 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.helper.DruidCoordinatorLogger - Server[172.19.0.8:8083, historical, _default_tier] has 0 left to load, 0 left to drop, 0 bytes queued, 0 bytes served.```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
