[GitHub] [spark] xuanyuanking commented on pull request #28904: [SPARK-30462][SS] Streamline the logic on file stream source and sink metadata log to avoid memory issue

xuanyuanking commented on pull request #28904: URL: https://github.com/apache/spark/pull/28904#issuecomment-659141316 `Well, I guess I already explained why compactLogs is the culprit of the memory issue, right? (#28904 (comment))` Yep that's right. I'm also looking at the code in de

[GitHub] [spark] xuanyuanking commented on pull request #28904: [SPARK-30462][SS] Streamline the logic on file stream source and sink metadata log to avoid memory issue

xuanyuanking commented on pull request #28904: URL: https://github.com/apache/spark/pull/28904#issuecomment-658803869 During searching the usage of `DELETE_ACTION`, I also found some usage for the thrid-party libraries of CompatibleFileStreamLog, e.g. elasticsearch-hadoop: https://githu

[GitHub] [spark] xuanyuanking commented on pull request #28904: [SPARK-30462][SS] Streamline the logic on file stream source and sink metadata log to avoid memory issue

xuanyuanking commented on pull request #28904: URL: https://github.com/apache/spark/pull/28904#issuecomment-655947410 retest this please This is an automated message from the Apache Git Service. To respond to the message, ple