Luke, my comment was that definition of "right thing" might be file-system dependent. For example, HDFS doesn't support overwriting existing files while other file systems might allow this. So we either have to pass the responsibility of handling that complexity and providing a unified interface to each FileSystem implementation or have to do that at the common FileSystems implementation. My argument was that it might be better to handle this complexity at a a single common place that is developed/maintained by the Beam team instead of passing that to FileSystem authors.
Tim, in the spirit of the above comment. I prefer handling "OVERWRITE_EXISTING_FILES" option at FileSystems level instead of passing to FileSystem interface. I believe most file-systems will either fail or support overwriting but will not offer that as an option and adding this to FileSystem interface will add an extra burden to each FileSystem author since this option might have to be supported by utilizing other methods of the FileSystem interface. I believe this complexity has to be pushed to FileSystems interface. Also, agree that this thread has become too long for a PR comment thread. [ Full content available at: https://github.com/apache/beam/pull/6289 ] This message was relayed via gitbox.apache.org for [email protected]
