BiggusDatus opened a new issue, #9604:
URL: https://github.com/apache/hop/issues/9604
### Apache Hop version?
2.20
### Java version?
openjdk version "25.0.5.1" 2026-11-17 LTS
### Operating system
Windows
### What happened?
Description
After upgrading from Apache Hop 2.17.0 to 2.20.0, accessing an S3 location
fails when the IAM user has s3:ListBucket permission restricted to a specific
prefix.
The same S3 location and credentials work correctly with Hop 2.17.0 and with
the AWS CLI.
Hop 2.20 appears to perform a ListBucket operation without the prefix
condition while resolving the S3 path. AWS therefore rejects the request before
Hop can access or write the requested object.
Environment
- Apache Hop 2.20.0
- Reproduced on Windows with Hop GUI
- Reproduced on Linux using the official apache/hop:latest Docker image
- AWS S3 region: eu-west-1
- Authentication: Access key and secret
- S3 access is intentionally restricted to a specific prefix
S3 structure
Example:
s3://example-bucket/allowed-prefix/
The IAM user is allowed to list/access objects under allowed-prefix/, but is
not allowed to list the complete bucket.
Steps to reproduce
1. Configure an IAM user with S3 access restricted to a specific bucket
prefix.
2. Verify that listing the allowed prefix works with AWS CLI:
aws s3api list-objects-v2 \
--bucket example-bucket \
--prefix "allowed-prefix/" \
--max-items 1
This succeeds.
1. Verify that listing the bucket without a prefix is intentionally denied:
aws s3api list-objects-v2 \
--bucket example-bucket \
--max-items 1
This returns AccessDenied for s3:ListBucket.
1. Configure the same credentials in an S3 Connection in Apache Hop 2.20.
2. Try to browse or access:
s3://example-bucket/allowed-prefix/
or use the equivalent path through the named S3 VFS connection.
Actual behavior
Hop 2.20 fails with:
AccessDeniedException: User ... is not authorized to perform: s3:ListBucket
on resource: "arn:aws:s3:::example-bucket"
because no identity-based policy allows the s3:ListBucket action
The relevant stack trace contains:
org.apache.hop.vfs.s3.s3common.S3CommonFileObject.resolveTypeViaList
org.apache.hop.vfs.s3.s3common.S3CommonFileObject.resolveTypeViaGetObject
org.apache.hop.vfs.s3.s3common.S3CommonFileObject.doAttach
When browsing the root of the named S3 connection (TEST:///), Hop also
attempts s3:ListAllMyBuckets. This permission is intentionally not granted to
the restricted IAM user.
Expected behavior
When accessing a path such as:
s3://example-bucket/allowed-prefix/
Hop should perform the ListBucket / ListObjectsV2 request with the
appropriate prefix (allowed-prefix/) instead of listing the bucket without a
prefix.
This would allow Hop to work with least-privilege IAM policies where
s3:ListBucket is restricted using an s3:prefix condition.
Hop should also ideally not require s3:ListAllMyBuckets when a specific
bucket or virtual path is already configured.
Additional observations
Using the exact same IAM user and credentials:
- AWS CLI list-objects-v2 with --prefix "allowed-prefix/": works
- AWS CLI list-objects-v2 without a prefix: fails with AccessDenied as
expected
- AWS CLI get-object for an object inside the allowed prefix: works
- Apache Hop 2.17.0 accessing/writing the same S3 location: works
- Apache Hop 2.20.0 accessing the same S3 location: fails
- Apache Hop 2.20.0 Parquet File Output to a local file: works
A workaround was found by setting the S3 Connection's virtualPath directly
in its metadata JSON to the permitted bucket/prefix. With the virtual path
configured, Hop is able to resolve more of the S3 hierarchy.
Example:
"virtualPath": "example-bucket/allowed-prefix/"
However, virtualPath does not appear to be exposed in the Hop 2.20 S3
Connection GUI.
Impact
This behavior makes upgrading from Hop 2.17 to Hop 2.20 problematic for
environments using least-privilege S3 IAM policies.
Granting unrestricted s3:ListBucket on the complete bucket or
s3:ListAllMyBuckets would broaden the IAM permissions solely to accommodate the
Hop VFS implementation.
Would it be possible for S3CommonFileObject to preserve the requested S3
prefix when performing its ListObjectsV2/ListBucket request?
### Issue Priority
Priority: 2
### Issue Component
Component: VFS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]