liutang123 opened a new issue #8690: KIS task fail when set segmentGranularity with timezone URL: https://github.com/apache/incubator-druid/issues/8690 ### Affected Version 0.15.1-incubating ### Description When submit a KIS Supervisor with segmentGranularity configured timezone, fowllow error will occurred. In task log: ```SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting. org.apache.druid.java.util.common.ISE: Could not allocate segment for row with timestamp[2019-10-16T20:12:11.000Z] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:605) [druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:246) [druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:167) [druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.15.1-incubating.jar:0.15.1-incubating] ``` In Overlord log: ``` LocalTaskActionClient - Performing action for task[index_kafka_bug_demo_57ebb7db959865d_bebmfiod]: SegmentAllocateAction{dataSource='bug_demo', timestamp=2019-10-16T20:12:11.000Z, queryGranularity={type=period, period=PT1M, timeZone=Asia/Shanghai, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=Asia/Shanghai, origin=null}, sequenceName='index_kafka_bug_demo_57ebb7db959865d_0', previousSegmentId='null', skipSegmentLineageCheck='true'} IndexerSQLMetadataStorageCoordinator - Cannot allocate new segment for dataSource[bug_demo], interval[2019-10-17T04:00:00.000+08:00/2019-10-17T05:00:00.000+08:00], maxVersion[2019-10-16T12:08:55.897Z]: conflicting segment[bug_demo_2019-10-16T20:00:00.000Z_2019-10-16T21:00:00.000Z_2019-10-16T12:08:55.897Z] ``` Follow config will reproduce the problem: ``` { "type": "kafka", "dataSchema": { "dataSource": "bug_demo", "parser": { "type": "string", "parseSpec": { "format": "json", "timestampSpec": { "column": "metricTime", "format": "yyyy-MM-dd HH:mm:ss" }, "dimensionsSpec": { "dimensions": [ "activity" ], "dimensionExclusions": [], "spatialDimensions": [] } } }, "metricsSpec": [ { "fieldName": "", "type": "count", "name": "numEvents" } ], "granularitySpec": { "type": "uniform", "segmentGranularity": { "type": "period", "period": "PT1H", "timeZone": "Asia/Shanghai" }, "queryGranularity": { "type": "period", "period": "PT1M", "timeZone": "Asia/Shanghai" } } }, "tuningConfig": { "type": "kafka", "maxRowsPerSegment": 5000000 }, "ioConfig": { "topic": "kafkaTopic", "consumerProperties": { XXXXXXXXXXXXXXXXX............ }, "taskCount": 1, "replicas": 1, "taskDuration": "PT10M" } } ``` In the metadata database, the peddingSegments table stores segmentId are as follows: ``` *************************** 1. row *************************** id: bug_demo_2019-10-17T04:00:00.000+08:00_2019-10-17T05:00:00.000+08:00_2019-10-16T12:08:55.897Z dataSource: bug_demo created_date: 2019-10-16T12:08:55.938Z start: 2019-10-17T04:00:00.000+08:00 end: 2019-10-17T05:00:00.000+08:00 sequence_name: index_kafka_gh2_test_liulijia_kis_origin_57b06c0cbaa83ea_0 ``` - Any debugging that you have already done 1. KIS task A start 2. KIS task A allocate pedding segment [dataSource: bug_demo, start: timeA, end: timeB] with segmentGranularity PT1H and time zone Asia/Shanghai. 3. Overlord receives request, and create ta new pendding segment with Asia/Shanghai timezone and insert it into metadata table. see [IndexerSQLMetadataStorageCoordinator.java#L720](https://github.com/apache/incubator-druid/blob/75527f09cddd5160e27d3af373c6a2bc656b6e7f/server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java#L720) and [IndexerSQLMetadataStorageCoordinator.java#L627](https://github.com/apache/incubator-druid/blob/75527f09cddd5160e27d3af373c6a2bc656b6e7f/server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java#L627) 4. KIS task A receive SegmentIdWithShardSpec, but its interval will deserialized as UTC, see [JodaStuff.java#L75](https://github.com/apache/incubator-druid/blob/75527f09cddd5160e27d3af373c6a2bc656b6e7f/processing/src/main/java/org/apache/druid/jackson/JodaStuff.java#L75) 5. KIS task A create segment bug_demo_timeA_timeB_versionS. timeA and timeB are UTC. 6. KIS task A stop after taskDuration reached. 7. KIS task B start 8. KIS task B allocate pedding segment [dataSource: bug_demo, start: timeA, end: timeB] with segmentGranularity PT1H and time zone Asia/Shanghai. 9. **bug 1** Overlord can not find segment bug_demo_timeA_timeB_versionS. 10. **bug 2** Overlord get pedding segment bug_demo_timeA_timeB_versionS but deserialized its interval as UTC, see [IndexerSQLMetadataStorageCoordinator.java#L163](https://github.com/apache/incubator-druid/blob/75527f09cddd5160e27d3af373c6a2bc656b6e7f/server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java#L163). Comparision of the two segments are not equal, see [IndexerSQLMetadataStorageCoordinator.java#L721](https://github.com/apache/incubator-druid/blob/75527f09cddd5160e27d3af373c6a2bc656b6e7f/server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java#L721) So, Should we set the pending segment to UTC?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
