I have tested HDFS (the whole point of this PR) and the performance is listed above (100 minutes drops to 42 minutes for a basic rewrite of 1.5TB avro file on a 10 node cluster).
GCS rename does a copy() and delete() internally, so should be the same runtime but I agree it should be confirmed as should local [ Full content available at: https://github.com/apache/beam/pull/6289 ] This message was relayed via gitbox.apache.org for [email protected]
