mcvsubbu commented on issue #4968: Remove Kafka and Stream dependency on avro URL: https://github.com/apache/incubator-pinot/pull/4968#issuecomment-573219019 > > > > Looks good. Minor comment. My preference is to add method in Schema class to get all the fieldspecs to extract. This would allow us to hide the incomingFieldSpec concept and also handle derived fields in future. > > > > I also want to deprecate timespec and move towards datetimefieldspec. > > > > > > > > > One reason NOT to do this in schema is because we will be creating a new collection that includes all fieldspecs except timefieldspec (which will be changed to be the incoming granularity spec). In the past, we have been careful about performance and garbage generation while ingesting records from high throughput streams (1k+ records per second per partition, peak at 10k when we are starting off consuming a new segment since we don't consume during segment build). > > > I suggest we leave it as it is. > > > > > > The other way of doing this will be to add a map of extractable field specs in `Schema`, and keep returning an immutable collection from that. Let me know if you want me to proceed in that direction. It will be a bit ugly since we need to make some assumptions that this API is called only after the schema is completely constructed. In that case, we will built it once and return the same objects all the time. > > @kishoreg > > I like that. Then the contract for decoders and readers will be to invoke, schema.getInputFieldsSpecs() or something like that. We can do this in another PR since it will need some discussion. In this PR, my suggestion is to keep RecordTranslator as an interface and probably move method to extract incoming time column to another base class or util or let each implementation take care of it. @kishoreg I already re-factored the RecordExtractor as an interface. If you can approve it, I can merge
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
