kishoreg commented on issue #9045:
URL: https://github.com/apache/pinot/issues/9045#issuecomment-1183324160
sorry for the delay in pitching in. I think the right solution might be to
support this at two levels
- Decoder (convert raw payload to GenericRow)
* Decoder should internally handle exception, retry etc but probably not
skip a record and return null instead or generic row with skip=true or
something like that and adhere to the contract.. return the new offset.
- Pinot (convert generic row to pinot data types)
* this should be common behavior across all decoders, if the conversion
fails and if the ignoreBadRows is set to true, we should be able to continue
with the next record else log the exception and stop the consumption
I also dont want to block @ddcprg. If you want to just handle the first part
as part of the decoder, should be fine. please make sure it still conforms to
the interface contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
