subject:"\[GitHub\] \[spark\] steveloughran commented on a change in pull request #31495\: \[SPARK\-34383\]\[SS\] Optimize WAL commit phase via reducing cost of filesystem operations"

[GitHub] [spark] steveloughran commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

2021-03-17 Thread GitBox

steveloughran commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r595923118 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CheckpointFileManager.scala ## @@ -277,7 +277,6 @@ class FileSystemB

[GitHub] [spark] steveloughran commented on a change in pull request #31495: [SPARK-34383][SS] Optimize WAL commit phase via reducing cost of filesystem operations

2021-02-08 Thread GitBox

steveloughran commented on a change in pull request #31495: URL: https://github.com/apache/spark/pull/31495#discussion_r571948537 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/HDFSMetadataLog.scala ## @@ -239,18 +239,35 @@ class HDFSMetadataLog