This is an automated email from the ASF dual-hosted git repository.
bamaer pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 414b851325 Update the VFS supported operations matrix and document
append on S3, Azure and Google Cloud Storage, fixes #9380 (#9384)
414b851325 is described below
commit 414b851325e3435f10798adc885fa2a6b14bc0cf
Author: Adalennis <[email protected]>
AuthorDate: Sun Nov 8 08:45:58 2026 +0100
Update the VFS supported operations matrix and document append on S3, Azure
and Google Cloud Storage, fixes #9380 (#9384)
---
docs/hop-user-manual/modules/ROOT/pages/vfs.adoc | 13 ++++++++++---
docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc | 7 +++++++
.../modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc | 7 +++++++
.../modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc | 7 +++++++
4 files changed, 31 insertions(+), 3 deletions(-)
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
index e17ef4f318..e7f49ce566 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs.adoc
@@ -321,12 +321,16 @@ Server-side restrictions (read-only mounts, bucket
policies, account permissions
|Scheme|Read|Write|Append|List|Create/Delete|Rename|Random read|Random
write|Last-modified
// Hop-managed providers
-|`s3` |✓|✓|✗|✓|✓|✓|✓|✗|read
-|`azure` |✓|✓|✗|✓|✓|✓|✗|✗|read
-|`gs` |✓|✓|✗|✓|✓|✗|✗|✗|read/set
+|`s3` |✓|✓|✓|✓|✓|✓|✓|✗|read
+|`azure` |✓|✓|✓|✓|✓|✓|✓|✗|read
+|`gs` |✓|✓|✓|✓|✓|✗|✓|✗|read/set
|`googledrive` |✓|✓|✗|✓|✓|✓|✓|✗|read
|`dropbox` |✓|✓|✗|✓|✓|✓|✗|✗|read
|`+<minio-conn>+` |✓|✓|✗|✓|✓|✓|✓|✗|read
+|`+<databricks-conn>+` |✓|✓|✗|✓|✓|✓|✗|✗|read
+|`+<git-conn>+` (see below) |✓|✓|✓|✓|✓|✓|✗|✗|read
+|`+<hdfs-conn>+` |✓|✓|✗|✓|✓|✓|✗|✗|read
+|`+<smb-conn>+` |✓|✓|✓|✓|✓|✓|✓|✓|read
|`webdav4`, `webdav4s`, named WebDAV connection |✓|✓|✗|✓|✓|✓|✓|✗|read
// Apache VFS providers
@@ -340,3 +344,6 @@ Server-side restrictions (read-only mounts, bucket
policies, account permissions
|`ftp`, `ftps` |✓|✓|✓|✓|✓|✓|✓|✗|read
|`sftp` |✓|✓|✓|✓|✓|✓|✓|✗|read/set
|===
+
+A Git connection reads files although it doesn't declare the read capability.
+A read-only Git connection, the default, refuses writes.
diff --git a/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
index f9bfa89244..f7844a839d 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/aws-s3-vfs.adoc
@@ -104,6 +104,13 @@ One connection to S3 is often limited to a few MB/s, so
reading many blocks at t
Text File Input running in parallel keeps up to copies × read-ahead blocks
requests in flight: 20 copies with 4 blocks each need 80 connections.
With fewer connections allowed than that, the other requests wait for a free
one.
+== Appending to a file
+
+Options such as *Append* in the Text file output transform add to an existing
object.
+S3 has no append operation, so Hop rebuilds the object with a multipart
upload: the existing content is copied on the server, and only the new bytes
are uploaded.
+An object smaller than 5 MiB is read back and uploaded again with the new
bytes.
+The existing content is only copied or read when the object hasn't changed
since Hop opened it for the append, otherwise the append fails.
+
== Usage and testing
To test if the configuration works you can simply upload a small CSV file in
an S3 bucket and then use File/Open in Hop GUI.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
index 2ac09700b8..89bde3c580 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
@@ -115,6 +115,13 @@ NOTE: A signature stops working once it expires, and Hop
reports the resulting f
authentication error. Renew the token, or use `Managed Identity` where you
want credentials that
rotate for you.
+== Appending to a file
+
+Options such as *Append* in the Text file output transform add to an existing
file.
+Hop adds the new bytes as extra blocks of the blob and commits them after the
existing ones, so the existing content is not uploaded again.
+A blob that was uploaded in a single request has no blocks yet, so its content
is uploaded again with the new bytes the first time.
+If the file changes while the append runs, the append fails instead of losing
data.
+
== Usage and testing
To test if the configuration works you can simply upload a small CSV file in
an Azure Storage folder and then use File/Open in Hop GUI.
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
index a36f46a495..6d953b49ee 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/google-cloud-storage-vfs.adoc
@@ -97,6 +97,13 @@ A write is never ended.
These messages are written to the general Hop log with the prefix `Google
Cloud Storage:`, for example in the console output of `hop-run`.
+== Appending to a file
+
+Options such as *Append* in the Text file output transform add to an existing
object.
+Google Cloud Storage objects can't be changed, so Hop writes the new bytes to
a temporary object, joins it to the end of the existing object with a compose
when the file is closed, and deletes the temporary object.
+If the object changes while the append runs, the append fails instead of
losing data.
+A composed object has no MD5 hash, only a CRC32C checksum.
+
== Usage and testing
To test if the configuration works type use File/Open in the GUI and type in
gs:// as a file location.