jerryshao commented on issue #179: Use Iceberg tables as sources for Spark Structured Streaming URL: https://github.com/apache/incubator-iceberg/issues/179#issuecomment-576647291 No, I will not store files directly in the offset json file, what I currently do is to store file paths, like what spark did to FileStreamSource, when the offset is loaded in, I will compare and filter out the files which are already processed. I'm currently still in prototyping, the content of `Offset` may be changed.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
