dbardbar commented on issue #12506: URL: https://github.com/apache/druid/issues/12506#issuecomment-1127961552
@a2l007 - Kafka does provide a way to re-read already ingested data, and Druid even has a pretty easy way to reset the Kafka marker so to re-read all data retained by Kafka. It is true that retention depends on exact configuration of Kafka, and if the user didn't configure it correctly, there may only a hours/days worth of data in Kafka. There's a feature offered by Confluent to make Kafka store/reload old data in S3, but AFAIK this is not part of vanilla Kafka. Doesn't it sound strange to you to make Druid solve the problem of long-term storage and retrieval of raw data? Why would it do a better job at this compared to Kafka? I don't know enough about Kinesis, so I have no idea if the a similar concept applies there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
