davsclaus opened a new pull request, #3206:
URL: https://github.com/apache/camel-kamelets/pull/3206

   Part of #3170: `aws-s3-event-based-source` can log in without static keys, 
through the IAM role of an EKS pod (IRSA or Pod Identity), an ECS task or an 
EC2 instance, an AWS profile, or session credentials. `aws-s3-sink` and 
`aws-s3-streaming-upload-sink` can encrypt the files with a KMS key and choose 
their storage class.
   
   ### The problem
   
   - `aws-s3-event-based-source` took only `accessKey` and `secretKey`, and 
both were required. Where there are no static keys, as with IAM roles, it could 
not be used.
   - Its `getObject` call put the keys into the URI without `RAW()`. A secret 
key with a `+` failed with `SignatureDoesNotMatch`, because the `+` was read as 
a space. The SQS side already used `RAW()`, so the queue worked and only the 
file fetch failed.
   - A bucket policy that requires SSE-KMS with a given key refused every 
upload of both sinks with `AccessDenied`. There was also no way to store the 
files in another storage class than `STANDARD`.
   
   ### The options
   
   `aws-s3-event-based-source`: the credential options of `aws-s3-source` and 
`aws-s3-sink`, with the same wording. They go to the SQS consumer and also to 
the internal `aws2-s3` `getObject` call. The new parameters are appended at the 
end, without reordering.
   
   | Option | SQS mapping | `getObject` URI (appended) |
   |---|---|---|
   | `useDefaultCredentialsProvider` | `useDefaultCredentialsProvider: 
"{{?useDefaultCredentialsProvider}}"` | 
`&useDefaultCredentialsProvider={{?useDefaultCredentialsProvider}}` |
   | `useProfileCredentialsProvider` | `useProfileCredentialsProvider: 
"{{?useProfileCredentialsProvider}}"` | 
`&useProfileCredentialsProvider={{?useProfileCredentialsProvider}}` |
   | `profileCredentialsName` | `profileCredentialsName: 
"{{?profileCredentialsName}}"` | 
`&profileCredentialsName={{?profileCredentialsName}}` |
   | `useSessionCredentials` | `useSessionCredentials: 
"{{?useSessionCredentials}}"` | 
`&useSessionCredentials={{?useSessionCredentials}}` |
   | `sessionToken` (secret) | `sessionToken: "RAW({{?sessionToken}})"` | 
`&sessionToken=RAW({{?sessionToken}})` |
   | `accessKey`, `secretKey` | unchanged | 
`accessKey={{accessKey}}&secretKey={{secretKey}}` becomes 
`accessKey=RAW({{?accessKey}})&secretKey=RAW({{?secretKey}})` |
   
   `accessKey` and `secretKey` are no longer in `required`, as in the other S3 
Kamelets.
   
   `aws-s3-sink` and `aws-s3-streaming-upload-sink`: appended to the end of the 
`aws2-s3` URI.
   
   | Option | Template mapping | What it does |
   |---|---|---|
   | `useAwsKMS` | `&useAwsKMS={{?useAwsKMS}}` | Encrypts each file with a KMS 
key (SSE-KMS). |
   | `awsKMSKeyId` | `&awsKMSKeyId={{?awsKMSKeyId}}` | The key, as its key ID 
or its ARN. Pattern `^[A-Za-z0-9:/_-]+$`. |
   | `useSSES3` | `&useSSES3={{?useSSES3}}` | Sends 
`x-amz-server-side-encryption: AES256` (SSE-S3), which some bucket policies 
require. |
   | `storageClass` | `&storageClass={{?storageClass}}` | The storage class, 
such as `STANDARD_IA` or `GLACIER_IR`. Pattern `^[A-Z_]+$`. |
   
   - **Defaults:** all new options are optional and have no `default`, as in 
#3174. When they are unset, the optional placeholders drop them. The endpoint 
URIs of the sinks and the SQS URI of the source are byte-identical to `main`'s. 
The `getObject` URI differs only by `RAW()` around the keys.
   - **Why `useSSES3`:** S3 encrypts new files with SSE-S3 anyway. Some bucket 
policies still deny a `PutObject` that does not ask for it in the header, and 
this option covers that. The component has no bucket-key option. SSE-C 
(`useCustomerKey`) is left out, because it needs the key and its MD5 in every 
request and in every read.
   - **The streaming sink:** aws2-s3 in `streamingUploadMode` uploads each file 
as a multipart upload. Its `CreateMultipartUpload` request gets the SSE 
settings and the storage class. This is shown by the policy case below and by 
`head-object`.
   - **Two component behaviours, now documented:**
     - `useAwsKMS` without `awsKMSKeyId` has no effect in `aws-s3-sink`: the 
simple put only sets SSE-KMS when a key ID is given. In the streaming sink it 
gives `aws:kms` with the key that AWS manages (`aws/s3`).
     - A `CamelAwsS3StorageClass` header on the message takes precedence over 
`storageClass`, in both sinks.
   - **Descriptions:**
     - The "Good to know" paragraphs point to the new options.
     - The event source's says that it logs in to the queue and the bucket with 
the same credentials.
     - `getObject` now says "with the same credentials as the queue".
     - The streaming sink's "Headers are not used" became "The names of the 
files come from `keyName`, not from headers", since the storage class header is 
used.
   - **Partial pages:**
     - The event source has a new authentication section: the four ways to log 
in, the default provider chain (including IRSA and Pod Identity), and an 
"Example with the IAM Role of the Pod".
     - Both sinks have an "Example with Encryption by a KMS Key and a Storage 
Class".
   - **From the AWS documentation, not tested here:** the `kms:GenerateDataKey` 
and `kms:Decrypt` permissions, the ARN being needed for a key of another 
account, and SSE-S3 being the default encryption of S3. What real AWS does with 
`useAwsKMS` together with `useSSES3` was not tested: the component then sends 
`AES256` together with the KMS key ID. On floci, SSE-S3 wins.
   
   ### Verification
   
   Tested with Camel JBang 4.24.0-SNAPSHOT against `floci/floci` 2.5.0 as S3, 
SQS, IAM, STS and KMS.
   
   - **Signatures:** floci was started with 
`FLOCI_AUTH_VALIDATE_SIGNATURES=true`, so a wrong secret key fails with 
`SignatureDoesNotMatch`. The keys were made with `iam create-access-key` until 
one had both `+` and `/` (`bmpgvQb+Nf9rrV0unR4/br0a/...`), and session 
credentials came from `sts get-session-token`.
   - **Bucket policy:** for the policy cases, a second floci ran with 
`FLOCI_SERVICES_IAM_ENFORCEMENT_ENABLED=true`. Its bucket policy denies 
`s3:PutObject` unless `s3:x-amz-server-side-encryption-aws-kms-key-id` is the 
given key ARN. The writer was an IAM user allowed `s3:*` and `kms:*`.
   - **Events:** the source read an EventBridge `Object Created` / `PutObject` 
event that was put on the queue by hand.
   - **Endpoint of `getObject`:** that internal call has no endpoint override, 
so the test pointed the `aws2-s3` component at floci with 
`camel.component.aws2-s3.overrideEndpoint`, `uriEndpointOverride` and 
`forcePathStyle`.
   - **How results were read:** `camel get endpoint --json` for the endpoint 
URIs, and `aws s3api head-object` for what floci recorded.
   
   **`aws-s3-event-based-source`**
   
   | Case | Result |
   |---|---|
   | `main`, `accessKey` + `secretKey` with `+` and `/`, `getObject: true` | 
SQS works; `getObject` fails: `S3Exception: The request signature we calculated 
does not match` (403) |
   | This branch, same keys | File fetched (`GOT key=hello.txt body=hello 
world`); `getObject` URI has `accessKey=RAW(...)&secretKey=RAW(...)` |
   | Options unset, `main` vs this branch | SQS and `kamelet:` endpoint URIs 
identical; the `getObject` URI differs only by `RAW()` around the two keys |
   | `main`, `useDefaultCredentialsProvider: true`, no keys | Fails at start: 
`mandatory parameters must be provided: accessKey, secretKey` |
   | `useDefaultCredentialsProvider: true`, keys in `AWS_ACCESS_KEY_ID` / 
`AWS_SECRET_ACCESS_KEY` | Queue read and file fetched; both URIs carry 
`useDefaultCredentialsProvider=true` and no keys |
   | `useDefaultCredentialsProvider: true`, credentials from a stand-in of the 
EKS Pod Identity agent (`AWS_CONTAINER_CREDENTIALS_FULL_URI` + 
`AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`, serving STS session credentials) | 
Queue read and file fetched; the agent got two requests with the token, one for 
the SQS client and one for the S3 client |
   | `useProfileCredentialsProvider: true`, `profileCredentialsName: kamtest`, 
credentials file with a wrong `[default]` and a right `[kamtest]` | Queue read 
and file fetched; both URIs carry 
`useProfileCredentialsProvider=true&profileCredentialsName=kamtest` |
   | The same without `profileCredentialsName` | Fails at start: `The security 
token included in the request is invalid` (the wrong `[default]` profile) |
   | `useSessionCredentials: true` + STS `accessKey`, `secretKey`, 
`sessionToken` | Queue read and file fetched; both URIs carry 
`sessionToken=RAW(...)&useSessionCredentials=true` |
   | The same with a bogus `sessionToken` | Fails at start: `The security token 
included in the request is invalid` |
   
   **`aws-s3-sink`** (`head-object` after one message)
   
   | Case | Result |
   |---|---|
   | Options unset, `main` vs this branch | Identical endpoint URIs; 
`StorageClass: STANDARD`, no SSE |
   | `useAwsKMS` + `awsKMSKeyId` (key ID) + `storageClass: STANDARD_IA` | 
`ServerSideEncryption: aws:kms`, `SSEKMSKeyId: <key id>`, `StorageClass: 
STANDARD_IA` |
   | `useAwsKMS` + `awsKMSKeyId` as a key ARN | `aws:kms`, `SSEKMSKeyId: 
arn:aws:kms:...:key/...` |
   | `useAwsKMS: true` without `awsKMSKeyId` | No SSE: the option has no effect 
(documented) |
   | `awsKMSKeyId` without `useAwsKMS` | No SSE |
   | `useSSES3: true` | `ServerSideEncryption: AES256` |
   | `useAwsKMS` + key + `useSSES3` | `AES256`, no key ID, on floci (documented 
as "do not combine") |
   | `storageClass: GLACIER` | `StorageClass: GLACIER`; `get-object` then fails 
with `InvalidObjectState`, as documented |
   | `storageClass: STANDARD_IA` and a `CamelAwsS3StorageClass: GLACIER` header 
| `StorageClass: GLACIER` (the header wins, documented) |
   | Bucket policy requiring the KMS key, `main` | `AccessDeniedException: User 
is not authorized to perform: s3:PutObject`; no file |
   | Bucket policy, this branch, `useAwsKMS` + key ARN | Uploaded; `aws:kms` 
with the key ARN |
   
   **`aws-s3-streaming-upload-sink`** (`batchMessageNumber: 2`, `head-object` 
of the first file)
   
   | Case | Result |
   |---|---|
   | Options unset, `main` vs this branch | Endpoint URIs identical apart from 
the test's `keyName`; `STANDARD`, no SSE, 24 bytes |
   | Bucket policy requiring the KMS key, `main` | Both messages fail with 
`AccessDeniedException` (on `CreateMultipartUpload`) |
   | Bucket policy, this branch, `useAwsKMS` + key ARN + `storageClass: 
STANDARD_IA` | Uploaded as a multipart upload (ETag `...-1`); `aws:kms`, key 
ARN, `STANDARD_IA`, 24 bytes |
   | `useAwsKMS: true` without `awsKMSKeyId` | `ServerSideEncryption: aws:kms`, 
no key ID (the AWS managed key, documented) |
   | `useSSES3: true` + `storageClass: GLACIER_IR` | `AES256`, `GLACIER_IR` |
   | `storageClass: STANDARD_IA` and a `CamelAwsS3StorageClass: GLACIER` header 
| `GLACIER` (the header wins, documented) |
   
   ### Build
   
   `./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed from 
the root. Before building, the library copies of the three Kamelets were 
refreshed with `./mvnw -pl library/camel-kamelets 
resources:copy-resources@copy-resource-one`. The build regenerated 
`kamelets-index.json`. `KameletsCatalogTest` needed no change.
   
   _Claude Code on behalf of Claus Ibsen_
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to