pchang388 commented on issue #12701: URL: https://github.com/apache/druid/issues/12701#issuecomment-1178207153
#3 Continued According to the source, a task can be in one of the states: * https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java ``` public enum Status { NOT_STARTED, STARTING, READING, PAUSED, PUBLISHING } ``` What we are seeing right now, that it is stuck in "STARTING" Status but appears to be still doing work on the peon side with segments and actions. This is what I see on the peon side during the 202 Accepted response * During the 202 Accepted, the main thing I can see but not sure what to make of it yet, is the appenderator-merge thread running that looks like it's actually working with the data, example: ``` 2022-07-06T21:56:34,597 INFO [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T20:00:00.000Z_2022-07-06T21:00:00.000Z_2022-07-06T20:08:32.626Z_48] of 41,979,452 bytes built from 11 incremental persist(s) in 16,246ms; pushed to deep storage in 11,819ms. Load spec is: {"type":"s3_zip","bucket":"REDACT","key":"druid/segments/REDACT/2022-07-06T20:00:00.000Z_2022-07-06T21:00:00.000Z/2022-07-06T20:08:32.626Z/48/998c542a-1af0-4d6d-8b47-0396eea79f30/index.zip","S3Schema":"s3n"} 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=0}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=1}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=2}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=3}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=4}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=5}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=6}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=7}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=8}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=9}] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.realtime.appenderator.StreamAppenderator - Segment[REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15] adding hydrant[FireHydrant{queryable=REDACT_2022-07-06T21:00:00.000Z_2022-07-06T22:00:00.000Z_2022-07-06T21:26:13.701Z_15, count=10}] 2022-07-06T21:56:34,601 INFO [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Indexes have incompatible dimension orders, try falling back on dimension ordering from ingestionSpec 2022-07-06T21:56:34,601 INFO [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Cannot fall back on dimension ordering from ingestionSpec as it does not exist 2022-07-06T21:56:34,601 WARN [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Indexes have incompatible dimension orders and there is no valid dimension ordering in the ingestionSpec, using lexicographic order. 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMergerV9 - Using SegmentWriteOutMediumFactory[TmpFileSegmentWriteOutMediumFactory] 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMergerV9 - Completed version.bin in 0 millis. 2022-07-06T21:56:34,601 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMergerV9 - Completed factory.json in 0 millis 2022-07-06T21:56:34,602 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [133,140] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,627 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[Name] conversions with cardinality[31,146] in 25 millis. 2022-07-06T21:56:34,628 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [133,900] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,653 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[ResourceId] conversions with cardinality[31,834] in 26 millis. 2022-07-06T21:56:34,653 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [156] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,653 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[ResourceKindKey] conversions with cardinality[4] in 0 millis. 2022-07-06T21:56:34,653 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [1,644] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,654 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[cpu] conversions with cardinality[49] in 1 millis. 2022-07-06T21:56:34,654 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [20,024] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,656 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[datastore] conversions with cardinality[1,581] in 2 millis. 2022-07-06T21:56:34,656 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [32] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,656 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[devices] conversions with cardinality[2] in 0 millis. 2022-07-06T21:56:34,657 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [88] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,657 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[disk] conversions with cardinality[2] in 1 millis. 2022-07-06T21:56:34,657 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [19,900] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,660 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[diskspace] conversions with cardinality[3,010] in 3 millis. 2022-07-06T21:56:34,660 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [40] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,660 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[gpu] conversions with cardinality[4] in 0 millis. 2022-07-06T21:56:34,660 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [18,496] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,663 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[guestfilesystem] conversions with cardinality[2,297] in 3 millis. 2022-07-06T21:56:34,663 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [108,276] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,683 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[hostname] conversions with cardinality[25,109] in 20 millis. 2022-07-06T21:56:34,684 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [16,776] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,685 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[metric] conversions with cardinality[463] in 1 millis. 2022-07-06T21:56:34,686 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [816] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,686 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[net] conversions with cardinality[27] in 1 millis. 2022-07-06T21:56:34,686 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [136] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,686 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[network] conversions with cardinality[5] in 0 millis. 2022-07-06T21:56:34,686 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[ping] conversions with cardinality[7] in 0 millis. 2022-07-06T21:56:34,686 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [640] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,686 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[snapshot] conversions with cardinality[55] in 0 millis. 2022-07-06T21:56:34,687 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [88] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,687 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[storageAdapter] conversions with cardinality[2] in 1 millis. 2022-07-06T21:56:34,687 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [1,192] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,687 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[template] conversions with cardinality[288] in 0 millis. 2022-07-06T21:56:34,687 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [2,944] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,688 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[virtualDisk] conversions with cardinality[89] in 1 millis. 2022-07-06T21:56:34,688 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMerger - Allocated [152] bytes of dictionary merging direct buffers 2022-07-06T21:56:34,688 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[REDACT_host] conversions with cardinality[7] in 0 millis. 2022-07-06T21:56:34,688 DEBUG [[index_kafka_REDACT_a5c10ee5effa63e_bhjndmoc]-appenderator-merge] org.apache.druid.segment.IndexMergerV9 - Completed dim conversions in 86 millis. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
