kty1965 commented on issue #23772: [SPARK-26866][SS] Support checkpoint with SubSequenceNumber to external/kinesis-asl URL: https://github.com/apache/spark/pull/23772#issuecomment-463212943 - I will re-open PR against master. - ok, change the title to [DSTREAM]. - we would like to use KPL(Kinesis Producer Library) which is optimize record size. But this library use called UserRecord which exetended Recrod has SubsequenceNumber. Kinesis-asl's checkpointer does not cotrol the subsequenceNumber, only sequence number. When re-submit spark stream application, processed duplicate record. We want to reduce this duplicate record.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
