winsmith opened a new issue, #15736:
URL: https://github.com/apache/druid/issues/15736
### Affected Version
28.0.1
### Description
I encountered an error during compaction and ingestion where tasks in middle
manager would crash randomly when ingesting a JSON line where the key contains
a comma (`,`). We call these "thorny" data because tasks that touch them crash.
Here's a line that crashes on ingest:
```json
{"receivedAt": "2024-01-20","A,B": "A"}
```
Here's an ingestion spec that triggers the error:
```json
{
"type": "index_parallel",
"spec": {
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "inline",
"data": "{\"receivedAt\": \"2024-01-20\",\"A,B\": \"C\"}\n"
},
"inputFormat": {
"type": "json"
}
},
"dataSchema": {
"dataSource": "telemetry-signals-thorny-test",
"timestampSpec": {
"column": "receivedAt",
"format": "iso"
},
"dimensionsSpec": {
"dimensionExclusions": [
"__time",
"count",
"receivedAt"
]
}
}
}
```
This results in task status `failed`:
```json
{
"id":
"index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
"groupId":
"index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
"type": "index_parallel",
"createdTime": "2024-01-22T14:37:20.082Z",
"queueInsertionTime": "1970-01-01T00:00:00.000Z",
"statusCode": "FAILED",
"status": "FAILED",
"runnerStatusCode": "WAITING",
"duration": 5685,
"location": {
"host": "10.244.7.175",
"port": 8102,
"tlsPort": -1
},
"dataSource": "telemetry-signals-thorny-test",
"errorMsg": "java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: org..."
}
```
Here's an excerpt from the index log in middle manager:
```
2024-01-22T14:37:24,925 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed
in-memory data for
segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z]
spill[0] to disk in [120] ms (1 rows).
2024-01-22T14:37:24,927 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed
in-memory data with commit metadata [null] for segments:
telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z
2024-01-22T14:37:24,927 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted
stats: processed rows: [1], persisted rows[1], sinks: [1], total fireHydrants
(across sinks): [1], persisted fireHydrants (across sinks): [1]
2024-01-22T14:37:24,928 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Preparing to
push (stats): processed rows: [1], sinks: [1], fireHydrants (across sinks): [1]
2024-01-22T14:37:24,934 WARN
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to
push merged index for
segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z].
org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in
line[A,B,0,187,436]
at
org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:89)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:550)
~[druid-processing-28.0.1.jar:28.0.1]
at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:205)
~[druid-processing-28.0.1.jar:28.0.1]
at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:195)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:913)
~[druid-server-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786)
~[druid-server-28.0.1.jar:28.0.1]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122)
~[guava-31.1-jre.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-01-22T14:37:24,941 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.IndexTask - Encountered exception in
BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number
of splits[5] in line[A,B,0,187,436]
at
org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1050)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:543)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1212)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:551)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
~[guava-31.1-jre.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
```
And here's the complete index log in the middle manager, just in case I
forgot something.
```
bash-5.1$ cat
index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14\:37\:20.080Z.log
2024-01-22T14:37:21,125 INFO [main]
org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator
6.2.5.Final
2024-01-22T14:37:21,839 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.8.0.jar,
druid-kafka-indexing-service-28.0.1.jar, zstd-jni-1.5.2-3.jar,
snappy-java-1.1.10.4.jar, kafka-clients-3.5.1.jar
2024-01-22T14:37:21,843 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-azure-extensions], jars:
azure-keyvault-core-1.0.0.jar, azure-storage-8.6.0.jar,
druid-azure-extensions-28.0.1.jar
2024-01-22T14:37:21,844 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [postgresql-metadata-storage], jars:
postgresql-metadata-storage-28.0.1.jar, checker-qual-3.31.0.jar,
postgresql-42.6.0.jar
2024-01-22T14:37:21,846 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-datasketches], jars: druid-datasketches-28.0.1.jar,
commons-math3-3.6.1.jar
2024-01-22T14:37:21,848 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-stats], jars: commons-math3-3.6.1.jar,
druid-stats-28.0.1.jar
2024-01-22T14:37:22,116 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-kafka-indexing-service], jars: lz4-java-1.8.0.jar,
druid-kafka-indexing-service-28.0.1.jar, zstd-jni-1.5.2-3.jar,
snappy-java-1.1.10.4.jar, kafka-clients-3.5.1.jar
2024-01-22T14:37:22,119 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-azure-extensions], jars:
azure-keyvault-core-1.0.0.jar, azure-storage-8.6.0.jar,
druid-azure-extensions-28.0.1.jar
2024-01-22T14:37:22,122 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [postgresql-metadata-storage], jars:
postgresql-metadata-storage-28.0.1.jar, checker-qual-3.31.0.jar,
postgresql-42.6.0.jar
2024-01-22T14:37:22,123 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-datasketches], jars: druid-datasketches-28.0.1.jar,
commons-math3-3.6.1.jar
2024-01-22T14:37:22,132 INFO [main] org.apache.druid.guice.ExtensionsLoader
- Loading extension [druid-stats], jars: commons-math3-3.6.1.jar,
druid-stats-28.0.1.jar
2024-01-22T14:37:23,384 INFO [main]
org.apache.druid.server.emitter.EmitterModule - Using emitter [NoopEmitter{}]
for metrics and alerts, with dimensions [{version=28.0.1}].
2024-01-22T14:37:23,714 WARN [main]
org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource
[telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you
sure this is what you want?
2024-01-22T14:37:23,747 INFO [main]
org.apache.druid.server.metrics.MetricsModule - Loaded 3 monitors:
org.apache.druid.query.ExecutorServiceMonitor,
org.apache.druid.curator.DruidConnectionStateListener,
org.apache.druid.server.initialization.jetty.JettyServerModule$JettyMonitor
2024-01-22T14:37:23,763 INFO [main] org.apache.druid.cli.CliPeon - Starting
up with processors [7], memory [2,147,483,648], maxMemory [3,221,225,472],
directMemory [17,179,869,184]. Properties follow.
2024-01-22T14:37:23,779 WARN [main]
org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode
argument value is null or not valid:[null], defaulting to[CLOSED_SEGMENTS]
2024-01-22T14:37:23,982 INFO [main]
org.apache.druid.query.DruidProcessingConfig - Detected max direct memory size
of [17,179,869,184] bytes
2024-01-22T14:37:24,000 INFO [main]
org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage
location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2024-01-22T14:37:24,050 INFO [main] org.apache.druid.guice.StorageNodeModule
- Segment cache not configured on ServerType [indexer-executor]. It will not be
assignable for segment placement
2024-01-22T14:37:24,108 INFO [main] org.eclipse.jetty.util.log - Logging
initialized @4005ms to org.eclipse.jetty.util.log.Slf4jLog
2024-01-22T14:37:24,121 INFO [main]
org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http
connector with port [8102]
2024-01-22T14:37:24,250 WARN [main]
org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is
inefficient for short content, break even is size 23
2024-01-22T14:37:24,256 INFO [main]
org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate
processing buffer[0] of size[7,000,000]
2024-01-22T14:37:24,262 INFO [main]
org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate
processing buffer[1] of size[7,000,000]
2024-01-22T14:37:24,266 INFO [main]
org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging
buffer[0] of size[7,000,000]
2024-01-22T14:37:24,270 INFO [main]
org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging
buffer[1] of size[7,000,000]
2024-01-22T14:37:24,333 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle
[module] stage [INIT]
2024-01-22T14:37:24,334 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle
[module] stage [NORMAL]
2024-01-22T14:37:24,338 INFO [main]
org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:zookeeper.version=3.5.10-a32c7183d42325b03e44a06aade6a0f16955bf13,
built on 05/29/2022 16:59 GMT
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:host.name=druid-druid-cluster-middlemanagers-1.druid-druid-cluster-middlemanagers.druid-main.svc.cluster.local
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:java.version=17.0.9
2024-01-22T14:37:24,344 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:java.vendor=Debian
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:java.home=/usr/lib/jvm/java-17-openjdk-amd64
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:java.class.path=/tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/commons-io-2.11.0.jar:lib/jboss-logging-3.4.1.Final.jar:lib/aws-java-sdk-core-1.12.497.jar:lib/derbynet-10.14.2.0.jar:lib/error_prone_annotations-2.20.0.jar:lib/jetty-continuation-9.4.53.v20231009.jar:lib/antlr4-runtime-4.5.3.jar:lib/guice-assistedinject-4.1.0.jar:lib/druid-indexing-hadoop-28.0.1.jar:lib/jakarta.inject-api-1.0.3.jar:lib/jetty-security-9.4.53.v20231009.jar:lib/jna-5.13.0.jar:lib/calcite-core-1.35.0.jar:lib/asm-commons-9.3.jar:lib/commons-compress-1.23.0.jar:lib/jakarta.activation-api-1.2.1.jar:lib/commons-codec-1.13.jar:lib/commons-dbcp2-2.0.1.jar:lib/jna-platform-5.13.0.jar:lib/druid-gcp-common-28.0.1.jar:lib/druid-indexing-service-28.0.1.jar:lib/google-oauth-client-1.34.1.jar:lib/resilience4j-core-1.3.1.jar:lib/config-magic-0.9.jar:lib/maven-repository-metadata-3.1.1.jar:lib/httpcore
-4.4.11.jar:lib/proj4j-1.2.2.jar:lib/aws-java-sdk-s3-1.12.497.jar:lib/xz-1.9.jar:lib/httpclient5-5.1.3.jar:lib/jaxb-api-2.3.1.jar:lib/jetty-util-ajax-9.4.53.v20231009.jar:lib/commons-compiler-3.1.9.jar:lib/druid-processing-28.0.1.jar:lib/jmespath-java-1.12.497.jar:lib/curator-framework-5.5.0.jar:lib/rhino-1.7.14.jar:lib/lz4-java-1.8.0.jar:lib/jersey-guice-1.19.4.jar:lib/aopalliance-1.0.jar:lib/google-http-client-jackson2-1.42.3.jar:lib/netty-codec-4.1.100.Final.jar:lib/gson-2.10.jar:lib/web-console-28.0.1.jar:lib/aws-java-sdk-kms-1.12.497.jar:lib/jetty-server-9.4.53.v20231009.jar:lib/commons-pool2-2.2.jar:lib/jackson-annotations-2.12.7.jar:lib/plexus-utils-3.0.24.jar:lib/elasticache-java-cluster-client-1.2.0.jar:lib/avatica-metrics-1.23.0.jar:lib/jetty-http-9.4.53.v20231009.jar:lib/aether-util-0.9.0.M2.jar:lib/airline-io-2.8.4.jar:lib/fastutil-core-8.5.4.jar:lib/maven-settings-3.1.1.jar:lib/disruptor-3.3.6.jar:lib/metrics-core-4.2.19.jar:lib/avatica-server-1.23.0.jar:lib/maven-setti
ngs-builder-3.1.1.jar:lib/fastutil-8.5.4.jar:lib/netty-codec-http-4.1.100.Final.jar:lib/jcl-over-slf4j-1.7.36.jar:lib/calcite-linq4j-1.35.0.jar:lib/guava-31.1-jre.jar:lib/jetty-util-9.4.53.v20231009.jar:lib/maven-artifact-3.6.0.jar:lib/grpc-context-1.27.2.jar:lib/sigar-1.6.5.132.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/json-simple-1.1.1.jar:lib/hibernate-validator-6.2.5.Final.jar:lib/jetty-io-9.4.53.v20231009.jar:lib/ion-java-1.0.2.jar:lib/jackson-jq-0.0.10.jar:lib/jersey-core-1.19.4.jar:lib/aether-spi-0.9.0.M2.jar:lib/netty-transport-native-epoll-4.1.100.Final-linux-x86_64.jar:lib/audience-annotations-0.5.0.jar:lib/httpclient-4.5.13.jar:lib/aggdesigner-algorithm-6.0.jar:lib/javax.activation-api-1.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/log4j-1.2-api-2.18.0.jar:lib/netty-transport-4.1.100.Final.jar:lib/log4j-api-2.18.0.jar:lib/jetty-client-9.4.53.v20231009.jar:lib/avatica-core-1.23.0.jar:lib/guice-4.1.0.jar:lib/jersey-server-1.19.4.jar:lib/druid-services-28.0.1.jar:lib/aws-ja
va-sdk-ec2-1.12.497.jar:lib/oshi-core-6.4.4.jar:lib/netty-handler-4.1.100.Final.jar:lib/jackson-datatype-guava-2.12.7.jar:lib/joni-2.1.27.jar:lib/jackson-module-jaxb-annotations-2.12.7.jar:lib/javax.servlet-api-3.1.0.jar:lib/opencsv-4.6.jar:lib/derby-10.14.2.0.jar:lib/google-http-client-apache-v2-1.42.3.jar:lib/vavr-match-0.10.2.jar:lib/asm-9.3.jar:lib/zstd-jni-1.5.2-3.jar:lib/asm-tree-9.3.jar:lib/jackson-jaxrs-smile-provider-2.12.7.jar:lib/jackson-jaxrs-base-2.12.7.jar:lib/log4j-core-2.18.0.jar:lib/vavr-0.10.2.jar:lib/ipaddress-5.3.4.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/checker-qual-2.5.7.jar:lib/jetty-servlet-9.4.53.v20231009.jar:lib/jts-io-common-1.19.0.jar:lib/datasketches-java-4.2.0.jar:lib/druid-server-28.0.1.jar:lib/jaxb-runtime-2.3.1.jar:lib/asm-analysis-9.3.jar:lib/jakarta.validation-api-2.0.2.jar:lib/datasketches-memory-2.2.0.jar:lib/joda-time-2.12.5.jar:lib/protobuf-java-3.24.0.jar:lib/accessors-smart-2.4.11.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/netty-resol
ver-dns-4.1.100.Final.jar:lib/jetty-rewrite-9.4.53.v20231009.jar:lib/httpcore5-5.1.3.jar:lib/janino-3.1.9.jar:lib/jackson-databind-2.12.7.jar:lib/google-api-client-2.2.0.jar:lib/curator-client-5.5.0.jar:lib/netty-common-4.1.100.Final.jar:lib/log4j-slf4j-impl-2.18.0.jar:lib/caffeine-2.8.0.jar:lib/stax-ex-1.8.jar:lib/airline-2.8.4.jar:lib/okhttp-1.0.2.jar:lib/jakarta.el-3.0.4.jar:lib/jts-core-1.19.0.jar:lib/compress-lzf-1.0.4.jar:lib/txw2-2.3.1.jar:lib/httpcore5-h2-5.1.3.jar:lib/FastInfoset-1.2.15.jar:lib/curator-recipes-5.5.0.jar:lib/netty-buffer-4.1.100.Final.jar:lib/jackson-dataformat-smile-2.12.7.jar:lib/apiguardian-api-1.1.2.jar:lib/jetty-servlets-9.4.53.v20231009.jar:lib/netty-codec-dns-4.1.100.Final.jar:lib/jackson-datatype-joda-2.12.7.jar:lib/maven-model-3.1.1.jar:lib/javax.activation-1.2.0.jar:lib/google-http-client-gson-1.42.3.jar:lib/slf4j-api-1.7.36.jar:lib/uzaygezen-core-0.2.jar:lib/jdbi-2.63.1.jar:lib/classmate-1.5.1.jar:lib/netty-codec-socks-4.1.100.Final.jar:lib/jetty-
proxy-9.4.53.v20231009.jar:lib/async-http-client-2.5.3.jar:lib/aether-api-0.9.0.M2.jar:lib/jackson-jaxrs-json-provider-2.12.7.jar:lib/javax.el-api-3.0.0.jar:lib/tesla-aether-0.0.5.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/jsr311-api-1.1.1.jar:lib/guice-multibindings-4.1.0.jar:lib/netty-reactive-streams-2.0.0.jar:lib/istack-commons-runtime-3.0.7.jar:lib/zookeeper-3.5.10.jar:lib/netty-3.10.6.Final.jar:lib/json-path-2.3.0.jar:lib/netty-resolver-4.1.100.Final.jar:lib/jcodings-1.0.43.jar:lib/commons-collections4-4.2.jar:lib/jersey-servlet-1.19.4.jar:lib/commons-lang-2.6.jar:lib/RoaringBitmap-0.9.49.jar:lib/guice-servlet-4.1.0.jar:lib/opencensus-api-0.31.1.jar:lib/commons-lang3-3.12.0.jar:lib/commons-math3-3.6.1.jar:lib/druid-sql-28.0.1.jar:lib/shims-0.9.49.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/druid-aws-common-28.0.1.jar:lib/netty-transport-native-unix-common-4.1.100.Final.jar:lib/aether-impl-0.9.0.M2.jar:lib/curator-x-discovery-5.5.0.jar:lib/cron-scheduler-0.1.jar:lib/ja
ckson-dataformat-cbor-2.12.7.jar:lib/zookeeper-jute-3.5.10.jar:lib/aws-java-sdk-sts-1.12.497.jar:lib/json-smart-2.4.11.jar:lib/maven-aether-provider-3.1.1.jar:lib/jackson-module-guice-2.12.7.jar:lib/jackson-core-2.12.7.jar:lib/commons-logging-1.1.1.jar:lib/netty-handler-proxy-4.1.100.Final.jar:lib/maven-model-builder-3.1.1.jar:lib/reactive-streams-1.0.2.jar:lib/icu4j-73.2.jar:lib/google-http-client-1.42.3.jar:lib/wagon-provider-api-2.4.jar:lib/opencensus-contrib-http-util-0.31.1.jar:lib/jsr305-2.0.1.jar:lib/failureaccess-1.0.1.jar:lib/log4j-jul-2.18.0.jar:lib/commons-collections-3.2.2.jar:lib/commons-beanutils-1.9.4.jar:lib/netty-transport-classes-epoll-4.1.100.Final.jar:lib/commons-text-1.10.0.jar:lib/plexus-interpolation-1.19.jar:lib/fastutil-extra-8.5.4.jar:lib/j2objc-annotations-1.3.jar:lib/javax.inject-1.jar:
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:java.io.tmpdir=/druid/data
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:java.compiler=<NA>
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:os.name=Linux
2024-01-22T14:37:24,345 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:os.arch=amd64
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:os.version=5.15.131.1-2.cm2
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:user.name=druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:user.home=/opt/druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:user.dir=/opt/druid
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:os.memory.free=1906MB
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:os.memory.max=3072MB
2024-01-22T14:37:24,346 INFO [main] org.apache.zookeeper.ZooKeeper - Client
environment:os.memory.total=2048MB
2024-01-22T14:37:24,349 INFO [main] org.apache.zookeeper.ZooKeeper -
Initiating client connection,
connectString=tiny-cluster-zk-0.tiny-cluster-zk,tiny-cluster-zk-1.tiny-cluster-zk,tiny-cluster-zk-2.tiny-cluster-zk
sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@731ab49b
2024-01-22T14:37:24,355 INFO [main] org.apache.zookeeper.common.X509Util -
Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable
client-initiated TLS renegotiation
2024-01-22T14:37:24,359 INFO [main] org.apache.zookeeper.ClientCnxnSocket -
jute.maxbuffer value is 4194304 Bytes
2024-01-22T14:37:24,367 INFO [main] org.apache.zookeeper.ClientCnxn -
zookeeper.request.timeout value is 0. feature enabled=
2024-01-22T14:37:24,374 INFO [main]
org.apache.curator.framework.imps.CuratorFrameworkImpl - Default schema
2024-01-22T14:37:24,380 INFO
[main-SendThread(tiny-cluster-zk-2.tiny-cluster-zk:2181)]
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
tiny-cluster-zk-2.tiny-cluster-zk/10.244.22.125:2181. Will not attempt to
authenticate using SASL (unknown error)
2024-01-22T14:37:24,391 INFO
[main-SendThread(tiny-cluster-zk-2.tiny-cluster-zk:2181)]
org.apache.zookeeper.ClientCnxn - Socket error occurred:
tiny-cluster-zk-2.tiny-cluster-zk/10.244.22.125:2181: Connection refused
2024-01-22T14:37:24,415 WARN [main]
org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource
[telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you
sure this is what you want?
2024-01-22T14:37:24,455 INFO [main]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Running with
task: {
"type" : "index_parallel",
"id" :
"index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
"groupId" :
"index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
"resource" : {
"availabilityGroup" :
"index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
"requiredCapacity" : 1
},
"spec" : {
"dataSchema" : {
"dataSource" : "telemetry-signals-thorny-test",
"timestampSpec" : {
"column" : "receivedAt",
"format" : "iso",
"missingValue" : null
},
"dimensionsSpec" : {
"dimensions" : [ ],
"dimensionExclusions" : [ "__time", "count", "receivedAt" ],
"includeAllDimensions" : false,
"useSchemaDiscovery" : false
},
"metricsSpec" : [ ],
"granularitySpec" : {
"type" : "uniform",
"segmentGranularity" : "DAY",
"queryGranularity" : {
"type" : "none"
},
"rollup" : true,
"intervals" : [ ]
},
"transformSpec" : {
"filter" : null,
"transforms" : [ ]
}
},
"ioConfig" : {
"type" : "index_parallel",
"inputSource" : {
"type" : "inline",
"data" : "{\"receivedAt\": \"2024-01-20\",\"A,B\": \"C\"}\n"
},
"inputFormat" : {
"type" : "json",
"keepNullColumns" : false,
"assumeNewlineDelimited" : false,
"useJsonNodeReader" : false
},
"appendToExisting" : false,
"dropExisting" : false
},
"tuningConfig" : {
"type" : "index_parallel",
"maxRowsPerSegment" : null,
"appendableIndexSpec" : {
"type" : "onheap",
"preserveExistingMetrics" : false
},
"maxRowsInMemory" : 1000000,
"maxBytesInMemory" : 0,
"skipBytesInMemoryOverheadCheck" : false,
"maxTotalRows" : null,
"numShards" : null,
"splitHintSpec" : null,
"partitionsSpec" : null,
"indexSpec" : {
"bitmap" : {
"type" : "roaring"
},
"dimensionCompression" : "lz4",
"stringDictionaryEncoding" : {
"type" : "utf8"
},
"metricCompression" : "lz4",
"longEncoding" : "longs"
},
"indexSpecForIntermediatePersists" : {
"bitmap" : {
"type" : "roaring"
},
"dimensionCompression" : "lz4",
"stringDictionaryEncoding" : {
"type" : "utf8"
},
"metricCompression" : "lz4",
"longEncoding" : "longs"
},
"maxPendingPersists" : 0,
"forceGuaranteedRollup" : false,
"reportParseExceptions" : false,
"pushTimeout" : 0,
"segmentWriteOutMediumFactory" : null,
"maxNumConcurrentSubTasks" : 1,
"maxRetry" : 3,
"taskStatusCheckPeriodMs" : 1000,
"chatHandlerTimeout" : "PT10S",
"chatHandlerNumRetries" : 5,
"maxNumSegmentsToMerge" : 100,
"totalNumMergeTasks" : 10,
"logParseExceptions" : false,
"maxParseExceptions" : 2147483647,
"maxSavedParseExceptions" : 0,
"maxColumnsToMerge" : -1,
"awaitSegmentAvailabilityTimeoutMillis" : 0,
"maxAllowedLockCount" : -1,
"partitionDimensions" : [ ]
}
},
"context" : {
"forceTimeChunkLock" : true,
"useLineageBasedSegmentAllocation" : true
},
"dataSource" : "telemetry-signals-thorny-test"
}
2024-01-22T14:37:24,457 INFO [main]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Attempting to
lock
file[/opt/druid/var/druid/task/slot1/index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z/lock].
2024-01-22T14:37:24,462 INFO [main]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Acquired lock
file[/opt/druid/var/druid/task/slot1/index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z/lock]
in 4ms.
2024-01-22T14:37:24,464 INFO [main]
org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk
lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,467 INFO [main]
org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage
location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2024-01-22T14:37:24,474 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Running task:
index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z
2024-01-22T14:37:24,476 WARN [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask
- Intervals are missing in granularitySpec while this task is potentially
overwriting existing segments. Forced to use timeChunk lock.
2024-01-22T14:37:24,477 WARN [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask
- firehoseFactory[InlineInputSource] is not splittable. Running sequentially.
2024-01-22T14:37:24,477 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle
[module] stage [SERVER]
2024-01-22T14:37:24,479 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk
lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,479 INFO [main] org.eclipse.jetty.server.Server -
jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git:
27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.9+9-Debian-1deb12u1
2024-01-22T14:37:24,480 WARN [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.IndexTask - Chat handler is already
registered. Skipping chat handler registration.
2024-01-22T14:37:24,489 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.IndexTask - Determining intervals and
shardSpecs
2024-01-22T14:37:24,499 INFO
[main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)]
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181. Will not attempt to
authenticate using SASL (unknown error)
2024-01-22T14:37:24,501 INFO
[main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)]
org.apache.zookeeper.ClientCnxn - Socket connection established, initiating
session, client: /10.244.7.175:43330, server:
tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181
2024-01-22T14:37:24,510 INFO
[main-SendThread(tiny-cluster-zk-0.tiny-cluster-zk:2181)]
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
tiny-cluster-zk-0.tiny-cluster-zk/10.244.2.104:2181, sessionid =
0x101bf1fb80d0233, negotiated timeout = 30000
2024-01-22T14:37:24,517 INFO [main-EventThread]
org.apache.curator.framework.state.ConnectionStateManager - State change:
CONNECTED
2024-01-22T14:37:24,530 INFO [main-EventThread]
org.apache.curator.framework.imps.EnsembleTracker - New config event received:
{}
2024-01-22T14:37:24,530 INFO [main-EventThread]
org.apache.curator.framework.imps.EnsembleTracker - New config event received:
{}
2024-01-22T14:37:24,538 WARN [NodeRoleWatcher[COORDINATOR]]
org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher
- Ignored event type [CONNECTION_RECONNECTED] for node watcher of role
[coordinator].
2024-01-22T14:37:24,538 WARN [NodeRoleWatcher[OVERLORD]]
org.apache.druid.curator.discovery.CuratorDruidNodeDiscoveryProvider$NodeRoleWatcher
- Ignored event type [CONNECTION_RECONNECTED] for node watcher of role
[overlord].
2024-01-22T14:37:24,549 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.IndexTask - Found intervals and
shardSpecs in 59ms
2024-01-22T14:37:24,549 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Using time chunk
lock since forceTimeChunkLock is [true] and mode is [REPLACE_LEGACY].
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[COORDINATOR]]
org.apache.druid.discovery.BaseNodeRoleWatcher - Node
[http://10.244.1.147:8088] of role [coordinator] detected.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[OVERLORD]]
org.apache.druid.discovery.BaseNodeRoleWatcher - Node
[http://10.244.1.150:8088] of role [overlord] detected.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[COORDINATOR]]
org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role
[coordinator] is now initialized with 1 nodes.
2024-01-22T14:37:24,562 INFO [NodeRoleWatcher[OVERLORD]]
org.apache.druid.discovery.BaseNodeRoleWatcher - Node watcher of role
[overlord] is now initialized with 1 nodes.
2024-01-22T14:37:24,696 INFO [main] org.eclipse.jetty.server.session -
DefaultSessionIdManager workerName=node0
2024-01-22T14:37:24,697 INFO [main] org.eclipse.jetty.server.session - No
SessionScavenger set, using defaults
2024-01-22T14:37:24,697 INFO [main] org.eclipse.jetty.server.session - node0
Scavenging every 600000ms
2024-01-22T14:37:24,721 WARN [task-runner-0-priority-0]
org.apache.druid.segment.indexing.DataSchema - Rollup is enabled for dataSource
[telemetry-signals-thorny-test] but no metricsSpec has been provided. Are you
sure this is what you want?
2024-01-22T14:37:24,771 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering
com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
2024-01-22T14:37:24,771 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering
com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class
2024-01-22T14:37:24,772 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering
org.apache.druid.server.initialization.jetty.CustomExceptionMapper as a
provider class
2024-01-22T14:37:24,772 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering
org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper as a
provider class
2024-01-22T14:37:24,772 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering
org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper as a
provider class
2024-01-22T14:37:24,772 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering
org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper
as a provider class
2024-01-22T14:37:24,772 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Registering
org.apache.druid.server.StatusResource as a root resource class
2024-01-22T14:37:24,774 INFO [main]
com.sun.jersey.server.impl.application.WebApplicationImpl - Initiating Jersey
application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
2024-01-22T14:37:24,799 INFO [task-runner-0-priority-0]
org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing
[1] segments in background
2024-01-22T14:37:24,802 INFO [task-runner-0-priority-0]
org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Pushing
segments:
[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z]
2024-01-22T14:37:24,805 INFO [task-runner-0-priority-0]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted
rows[1] and (estimated) bytes[154]
2024-01-22T14:37:24,837 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.initialization.jetty.CustomExceptionMapper to
GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,839 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper to
GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,839 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper to
GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,840 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper
to GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,840 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to
GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,848 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to
GuiceManagedComponentProvider with the scope "Singleton"
2024-01-22T14:37:24,925 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed
in-memory data for
segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z]
spill[0] to disk in [120] ms (1 rows).
2024-01-22T14:37:24,927 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Flushed
in-memory data with commit metadata [null] for segments:
telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z
2024-01-22T14:37:24,927 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-persist]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Persisted
stats: processed rows: [1], persisted rows[1], sinks: [1], total fireHydrants
(across sinks): [1], persisted fireHydrants (across sinks): [1]
2024-01-22T14:37:24,928 INFO
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Preparing to
push (stats): processed rows: [1], sinks: [1], fireHydrants (across sinks): [1]
2024-01-22T14:37:24,934 WARN
[[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]-appenderator-merge]
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to
push merged index for
segment[telemetry-signals-thorny-test_2024-01-20T00:00:00.000Z_2024-01-21T00:00:00.000Z_2024-01-22T14:37:24.693Z].
org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in
line[A,B,0,187,436]
at
org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:89)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:550)
~[druid-processing-28.0.1.jar:28.0.1]
at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:205)
~[druid-processing-28.0.1.jar:28.0.1]
at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:195)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:913)
~[druid-server-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786)
~[druid-server-28.0.1.jar:28.0.1]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122)
~[guava-31.1-jre.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-01-22T14:37:24,941 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.common.task.IndexTask - Encountered exception in
BUILD_SEGMENTS.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number
of splits[5] in line[A,B,0,187,436]
at
org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:1050)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.IndexTask.runTask(IndexTask.java:543)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runSequential(ParallelIndexSupervisorTask.java:1212)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.runTask(ParallelIndexSupervisorTask.java:551)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:178)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:478)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:450)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
~[guava-31.1-jre.jar:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: org.apache.druid.java.util.common.ISE: Wrong number
of splits[5] in line[A,B,0,187,436]
at
com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:588)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:567)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91)
~[guava-31.1-jre.jar:?]
at
org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAndClear(BatchAppenderatorDriver.java:151)
~[druid-server-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.pushAllAndClear(BatchAppenderatorDriver.java:134)
~[druid-server-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.InputSourceProcessor.process(InputSourceProcessor.java:131)
~[druid-indexing-service-28.0.1.jar:28.0.1]
at
org.apache.druid.indexing.common.task.IndexTask.generateAndPublishSegments(IndexTask.java:940)
~[druid-indexing-service-28.0.1.jar:28.0.1]
... 13 more
Caused by: java.lang.RuntimeException:
org.apache.druid.java.util.common.ISE: Wrong number of splits[5] in
line[A,B,0,187,436]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:998)
~[druid-server-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786)
~[druid-server-28.0.1.jar:28.0.1]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122)
~[guava-31.1-jre.jar:?]
... 3 more
Caused by: org.apache.druid.java.util.common.ISE: Wrong number of splits[5]
in line[A,B,0,187,436]
at
org.apache.druid.java.util.common.io.smoosh.SmooshedFileMapper.load(SmooshedFileMapper.java:89)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.java.util.common.io.smoosh.Smoosh.map(Smoosh.java:32)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:550)
~[druid-processing-28.0.1.jar:28.0.1]
at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:205)
~[druid-processing-28.0.1.jar:28.0.1]
at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:195)
~[druid-processing-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:913)
~[druid-server-28.0.1.jar:28.0.1]
at
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:786)
~[druid-server-28.0.1.jar:28.0.1]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:250)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.doTransform(AbstractTransformFuture.java:240)
~[guava-31.1-jre.jar:?]
at
com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:122)
~[guava-31.1-jre.jar:?]
... 3 more
2024-01-22T14:37:24,948 WARN [task-runner-0-priority-0]
org.apache.druid.segment.realtime.firehose.ServiceAnnouncingChatHandlerProvider
-
handler[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z]
not currently registered, ignoring.
2024-01-22T14:37:24,950 INFO [task-runner-0-priority-0]
org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed
with status: {
"id" :
"index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z",
"status" : "FAILED",
"duration" : 474,
"errorMsg" : "java.lang.RuntimeException:
java.util.concurrent.ExecutionException: java.lang.RuntimeException: org...",
"location" : {
"host" : null,
"port" : -1,
"tlsPort" : -1
}
}
2024-01-22T14:37:25,157 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.http.security.StateResourceFilter to
GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,181 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.http.SegmentListerResource to
GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,189 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.QueryResource to GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,190 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.segment.realtime.firehose.ChatHandlerResource to
GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,193 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.http.security.ConfigResourceFilter to
GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,197 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.query.lookup.LookupListeningResource to
GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,199 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.query.lookup.LookupIntrospectionResource to
GuiceInstantiatedComponentProvider
2024-01-22T14:37:25,200 INFO [main]
com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding
org.apache.druid.server.StatusResource to GuiceManagedComponentProvider with
the scope "Singleton"
2024-01-22T14:37:25,221 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler - Started
o.e.j.s.ServletContextHandler@573f2e67{/,null,AVAILABLE}
2024-01-22T14:37:25,232 INFO [main]
org.eclipse.jetty.server.AbstractConnector - Started
ServerConnector@69fe8c75{HTTP/1.1, (http/1.1)}{0.0.0.0:8102}
2024-01-22T14:37:25,233 INFO [main] org.eclipse.jetty.server.Server -
Started @5130ms
2024-01-22T14:37:25,233 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle
[module] stage [ANNOUNCEMENTS]
2024-01-22T14:37:25,233 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started
lifecycle [module]
2024-01-22T14:37:25,252 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [ANNOUNCEMENTS]
2024-01-22T14:37:25,255 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [SERVER]
2024-01-22T14:37:25,261 INFO [main]
org.eclipse.jetty.server.AbstractConnector - Stopped
ServerConnector@69fe8c75{HTTP/1.1, (http/1.1)}{0.0.0.0:8102}
2024-01-22T14:37:25,261 INFO [main] org.eclipse.jetty.server.session - node0
Stopped scavenging
2024-01-22T14:37:25,262 INFO [main]
org.eclipse.jetty.server.handler.ContextHandler - Stopped
o.e.j.s.ServletContextHandler@573f2e67{/,null,STOPPED}
2024-01-22T14:37:25,291 INFO [main]
org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting
graceful shutdown of
task[index_parallel_telemetry-signals-thorny-test_deolknmp_2024-01-22T14:37:20.080Z].
2024-01-22T14:37:25,292 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [NORMAL]
2024-01-22T14:37:25,307 INFO
[LookupExtractorFactoryContainerProvider-MainThread]
org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop
exited. Lookup notices are not handled anymore.
2024-01-22T14:37:25,315 INFO [Curator-Framework-0]
org.apache.curator.framework.imps.CuratorFrameworkImpl -
backgroundOperationsLoop exiting
2024-01-22T14:37:25,422 INFO [main-EventThread]
org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
0x101bf1fb80d0233
2024-01-22T14:37:25,422 INFO [main] org.apache.zookeeper.ZooKeeper -
Session: 0x101bf1fb80d0233 closed
2024-01-22T14:37:25,434 INFO [main]
org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle
[module] stage [INIT]
Finished peon task
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
