abhishekagarwal87 commented on issue #12506:
URL: https://github.com/apache/druid/issues/12506#issuecomment-1129617650
Thank you for this proposal, Atul. I have similar sentiment as @dbardbar
here. We are trying to use druid to do something it is not meant for. The
proposed extension targets a very particular/specific problem and is unlikely
to be useful beyond those few use cases. I am afraid that if we do build this
extension, it will not get regular maintenance down the road.
Thinking out loud does a setup like the below work?
- There is an ingestion job that ingests data from Kafka topic to a `raw`
datasource. This ingestion doesn't use rollup and ingests all the raw data as
it is and packs it into segments.
- The `raw` datasource is not stored on historicals and only on deep
storage. This can be configured via load rules.
- There is also a regular ingestion pipeline that is ingesting data with
rollup etc into the `queryable` datasource.
- If the rollup spec is wrong and the `queryable` datasource needs to be
fixed, you can re-index data from the `raw` datasource to the `queryable`
datasource.
Also, the current license with S3 Sink only aims to deter competitors and
regular users can still use it for free. I have used it myself a few years ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
