ChrisSamo632 commented on pull request #4822: URL: https://github.com/apache/nifi/pull/4822#issuecomment-804689864
> @ChrisSamo632 - out of curiosity, did you consider the addition of a record reader/writer like with ConsumeKafkaRecord processors? At the moment, as far as I can tell, we would have one record per flow file. Using the records abstraction would provide options such as schema validation, format transformation, as well as having multiple records in one flow file (which is greatly improving the performances in case there is a high number of messages per second). Think I considered this in our original slack conversation, but thought I'd leave it record writers at the minute for simplicity and to better understand how the Kinesis message (vs. Record) structure works With the KCL worker and multi-threaded consumer approach taken by the KCL library, we'd need a way of combining the records in the processor too (how do we combine records from different consumers? How about where the consumers are reading from different shards?) So I thought a record reader may be a sensible extension (fully agree it would be good to include from a nifi perspective) once there's more understanding of how people use the processor and how it works with different kinesis setups (so far I've only really tested it with simple streams) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
