amishasthana opened a new issue #9133: Kinesis Indexing throwing exception URL: https://github.com/apache/druid/issues/9133 I am using Druid and druid kinesis plugin to get the kinesis stream data into my druid instance. I am running "apache-druid-0.16.1-incubating" version. Server started on local machine as micro service quick start. The Kinesis plugin has been enabled. I create a Json template, like this and create the Data store for kinesis. { "type": "kinesis", "dataSchema": { "dataSource": "<A data source name>", "parser": { "type": "string", "parseSpec": { "format": "json", "timestampSpec": { "column": "<a column of type timestamp>", "format": "auto" }, "dimensionsSpec": { "dimensions": [], "dimensionExclusions": [ "timestamp" ] } } }, "granularitySpec": { "type": "uniform", "segmentGranularity": "HOUR", "queryGranularity": "NONE" } }, "tuningConfig": { "type": "kinesis", "maxRowsPerSegment": 5000000, "resetOffsetAutomatically": "true", "useEarliestOffset" : "false" }, "ioConfig": { "stream": "stream-name which exist in AWS", "endpoint": "kinesis.us-east-1.amazonaws.com", "taskCount": 1, "replicas": 1, "taskDuration": "PT1H", "recordsPerFetch": 2000, "fetchDelayMillis": 1000 } } The data source gets created. I have added the following aws key/secret key in file "conf/druid/single-server/micro-quickstart/_common/common.runtime.properties". druid.kinesis.accessKey=<The key> druid.kinesis.secretKey=<The secret key> Do note that with these values I can connect from this local machine to said AWS kinesis stream using KCL classes. When I start I see the following exceptions: ### 2020-01-06T21:17:03,581 WARN [KinesisSupervisor-am-kinesis-matched-v7] org.apache.druid.indexing.kinesis.KinesisRecordSupplier - timed out while trying to fetch position for shard[shardId-000000000000], likely no more records in shard 2020-01-06T21:17:03,587 WARN [KinesisSupervisor-am-kinesis-matched-v7] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Exception in supervisor run loop for dataSource [am-kinesis-matched-v7] org.apache.druid.java.util.common.ISE: unable to fetch sequence number for partition[shardId-000000000000] from stream at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.getOffsetFromStorageForPartition(SeekableStreamSupervisor.java:2559) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.generateStartingSequencesForPartitionGroup(SeekableStreamSupervisor.java:2499) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.createNewTasks(SeekableStreamSupervisor.java:2397) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.runInternal(SeekableStreamSupervisor.java:1066) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor$RunNotice.handle(SeekableStreamSupervisor.java:293) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating] at org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor.lambda$tryInit$3(SeekableStreamSupervisor.java:749) ~[druid-indexing-service-0.16.1-incubating.jar:0.16.1-incubating] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232] One another point to note, when there are new records in stream, I do see that Druid task is getting initiated. In other words it seem to be connected and aware of changes in kinesis stream.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
