wenxuanguan edited a comment on issue #25618: [SPARK-28908][SS]Implement Kafka EOS sink for Structured Streaming URL: https://github.com/apache/spark/pull/25618#issuecomment-526422493 > Before reviewing the design, I may need to say, you are encouraged to at least mention it if you borrow the code from somewhere, so that we are sure that there's no license issue, even no license issue, at least they've got credit. > > https://github.com/apache/spark/pull/25618/files#diff-c1e1dbc4a986c69ef54e1eebe880d4e9 > https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.11/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/FlinkKafkaProducer.java @HeartSaVioR Thanks for your advice. Indeed I refer to the flink FlinkKafkaProducer.java to resume transaction since there is no kafka API to support this function. And I will add it to annotation and design sketch. About the design of 2PC, it is a common way to implement EOS and I think spark hdfs sink is also apply with it, and that is what I referred.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
