jerryshao commented on issue #179: Use Iceberg tables as sources for Spark Structured Streaming URL: https://github.com/apache/incubator-iceberg/issues/179#issuecomment-576577519 @aokolnychyi I've already started a simple workable version of streaming read, hopes that I can share it soon. About batch size control, what I did currently is to control the number of files read per batch, and record them as a Spark `Offset`, so that Spark could continue from restart. I haven't sorted out all the details, I would start building a simple version first, and then refine all the details.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
