Hey there, With the recent improvements to the Kafka Indexing Service we've been migrating over from Tranquility and have had a very positive experience.
However one of the downsides to using the KIS, is that the number of segments generated for each period can't be smaller than the number of tasks required to consume the queue. So if you have a use case involving ingesting from a topic with a high rate of large messages but your spec only extracts a small proportion of fields you may be forced to run a large number of tasks that generate very small segments. This email is to check in for peoples thoughts on separating consuming and parsing messages from indexing and segment management, in a similar fashion to how Tranquility operates. Potentially - we could have the supervisor spawn two types of task that can be configured independently, a consumer and an appender. The consumer would parse the message based on the spec and then pass the results to the appropriate appender task which builds the segment. Another advantage to this approach is that it would allow creating multiple datasources from a single consumer group rather than ingesting the same topic multiple times. I'm quite new to the codebase so all thoughts and comments are welcome! Best regards, Dylan
