BiggusDatus opened a new issue, #9604:
URL: https://github.com/apache/hop/issues/9604

   ### Apache Hop version?
   
   2.20
   
   ### Java version?
   
   openjdk version "25.0.5.1" 2026-11-17 LTS
   
   ### Operating system
   
   Windows
   
   ### What happened?
   
   
   Description
   After upgrading from Apache Hop 2.17.0 to 2.20.0, accessing an S3 location 
fails when the IAM user has s3:ListBucket permission restricted to a specific 
prefix.
   The same S3 location and credentials work correctly with Hop 2.17.0 and with 
the AWS CLI.
   Hop 2.20 appears to perform a ListBucket operation without the prefix 
condition while resolving the S3 path. AWS therefore rejects the request before 
Hop can access or write the requested object.
   Environment
   
   - Apache Hop 2.20.0
   - Reproduced on Windows with Hop GUI
   - Reproduced on Linux using the official apache/hop:latest Docker image
   - AWS S3 region: eu-west-1
   - Authentication: Access key and secret
   - S3 access is intentionally restricted to a specific prefix
   S3 structure
   Example:
   s3://example-bucket/allowed-prefix/
   The IAM user is allowed to list/access objects under allowed-prefix/, but is 
not allowed to list the complete bucket.
   Steps to reproduce
   
   1. Configure an IAM user with S3 access restricted to a specific bucket 
prefix.
   2. Verify that listing the allowed prefix works with AWS CLI:
   aws s3api list-objects-v2 \
     --bucket example-bucket \
     --prefix "allowed-prefix/" \
     --max-items 1
   This succeeds.
   
   1. Verify that listing the bucket without a prefix is intentionally denied:
   aws s3api list-objects-v2 \
     --bucket example-bucket \
     --max-items 1
   This returns AccessDenied for s3:ListBucket.
   
   1. Configure the same credentials in an S3 Connection in Apache Hop 2.20.
   2. Try to browse or access:
   s3://example-bucket/allowed-prefix/
   or use the equivalent path through the named S3 VFS connection.
   Actual behavior
   Hop 2.20 fails with:
   AccessDeniedException: User ... is not authorized to perform: s3:ListBucket
   on resource: "arn:aws:s3:::example-bucket"
   because no identity-based policy allows the s3:ListBucket action
   The relevant stack trace contains:
   org.apache.hop.vfs.s3.s3common.S3CommonFileObject.resolveTypeViaList
   org.apache.hop.vfs.s3.s3common.S3CommonFileObject.resolveTypeViaGetObject
   org.apache.hop.vfs.s3.s3common.S3CommonFileObject.doAttach
   When browsing the root of the named S3 connection (TEST:///), Hop also 
attempts s3:ListAllMyBuckets. This permission is intentionally not granted to 
the restricted IAM user.
   Expected behavior
   When accessing a path such as:
   s3://example-bucket/allowed-prefix/
   Hop should perform the ListBucket / ListObjectsV2 request with the 
appropriate prefix (allowed-prefix/) instead of listing the bucket without a 
prefix.
   This would allow Hop to work with least-privilege IAM policies where 
s3:ListBucket is restricted using an s3:prefix condition.
   Hop should also ideally not require s3:ListAllMyBuckets when a specific 
bucket or virtual path is already configured.
   Additional observations
   Using the exact same IAM user and credentials:
   
   - AWS CLI list-objects-v2 with --prefix "allowed-prefix/": works
   - AWS CLI list-objects-v2 without a prefix: fails with AccessDenied as 
expected
   - AWS CLI get-object for an object inside the allowed prefix: works
   - Apache Hop 2.17.0 accessing/writing the same S3 location: works
   - Apache Hop 2.20.0 accessing the same S3 location: fails
   - Apache Hop 2.20.0 Parquet File Output to a local file: works
   A workaround was found by setting the S3 Connection's virtualPath directly 
in its metadata JSON to the permitted bucket/prefix. With the virtual path 
configured, Hop is able to resolve more of the S3 hierarchy.
   Example:
   "virtualPath": "example-bucket/allowed-prefix/"
   However, virtualPath does not appear to be exposed in the Hop 2.20 S3 
Connection GUI.
   Impact
   This behavior makes upgrading from Hop 2.17 to Hop 2.20 problematic for 
environments using least-privilege S3 IAM policies.
   Granting unrestricted s3:ListBucket on the complete bucket or 
s3:ListAllMyBuckets would broaden the IAM permissions solely to accommodate the 
Hop VFS implementation.
   Would it be possible for S3CommonFileObject to preserve the requested S3 
prefix when performing its ListObjectsV2/ListBucket request?
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: VFS


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to