hililiwei commented on PR #7638: URL: https://github.com/apache/iceberg/pull/7638#issuecomment-1554342314
> I am not sure this is a fair comparison. Flink filesystem connector is storing files on distributed file system (like S3) directly. there is no table format abstraction. hence success file is the only option. When we process data, we often have not only streaming jobs, but also many Spark batch jobs behind the streaming jobs. Flink+Hive+Spark is a very common combination. Similarly, when we switch Hive to Iceberg, we need the streaming job to tell the scheduling system when to start the Spark task. We use FileSystem very rarely. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
