Re: [PR] WiP: AWS: S3SignerServlet should strip out more request headers for caching [iceberg]

2026-03-04 Thread via GitHub 


steveloughran commented on PR #15428:
URL: https://github.com/apache/iceberg/pull/15428#issuecomment-3998986592

   FWIW there's also an x-amz-te which comes with put requests. Ignoring those 
as it is GET and HEAD which are cacheable.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] WiP: AWS: S3SignerServlet should strip out more request headers for caching [iceberg]

2026-03-03 Thread via GitHub 


steveloughran commented on PR #15428:
URL: https://github.com/apache/iceberg/pull/15428#issuecomment-3992372880

   FWIW, headers on requests picked up by running some cloudstore commands 
against s3 london, s3a client set to use sse-kms encryption. The bucket is 
versioned so on repeated reads the if-match header would be used to declare the 
version.
   
   ## HEAD /f
   ```
   2026-03-03 16:22:07,956 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "HEAD /f HTTP/1.1[\r][\n]"
   2026-03-03 16:22:07,956 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Host: stevel-london.s3.eu-west-2.amazonaws.com[\r][\n]"
   2026-03-03 16:22:07,956 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "amz-sdk-invocation-id: 
a31481a9-7186-008d-b3b3-ea125702f45a[\r][\n]"
   2026-03-03 16:22:07,956 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "amz-sdk-request: attempt=1; max=3[\r][\n]"
   2026-03-03 16:22:07,956 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Authorization: AWS4-HMAC-SHA256 
Credential=AKI7/20260303/eu-west-2/s3/aws4_request, 
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;host;referer;x-amz-content-sha256;x-amz-date,
 Signature=1dc..b[\r][\n]"
   2026-03-03 16:22:07,956 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Referer: 
https://audit.example.org/hadoop/1/op_delete/4022a50f-dd44-4915-b6ec-85c6cb7337c5-0026/?op=op_delete&p1=s3a://stevel-london/f&pr=stevel&ps=0b35c845-dbf9-4a18-9edd-61469b1f3644&cm=StoreDiag&id=4022a50f-dd44-4915-b6ec-85c6cb7337c5-0026&t0=1&fs=4022a50f-dd44-4915-b6ec-85c6cb7337c5&t1=1&ts=1772554927951[\r][\n]";
   2026-03-03 16:22:07,957 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "User-Agent: Hadoop 3.4.3 aws-sdk-java/2.35.4 md/io#sync 
md/http#Apache ua/2.1 api/S3#2.35.x os/Mac_OS_X#26.3 lang/java#17.0.17 
md/OpenJDK_64-Bit_Server_VM#17.0.17+10-LTS md/vendor#Amazon.com_Inc. md/en_GB 
m/F,G hll/cross-region[\r][\n]"
   2026-03-03 16:22:07,957 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
   2026-03-03 16:22:07,957 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "X-Amz-Date: 20260303T162207Z[\r][\n]"
   2026-03-03 16:22:07,957 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]"
   2026-03-03 16:22:07,957 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "[\r][\n]"
   ```
   
   ## PUT of directory marker (not relevant to s3file IO). adds 
x-amz-server-side-encryption x-amz-server-side-encryption-aws-kms-key-id, 
Content-Length, Content-Type.
   
   
   
   ```
   2026-03-03 16:22:07,852 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "PUT /f/ HTTP/1.1[\r][\n]"
   2026-03-03 16:22:07,852 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Host: stevel-london.s3.eu-west-2.amazonaws.com[\r][\n]"
   2026-03-03 16:22:07,852 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "amz-sdk-invocation-id: 
8473b1fd-3959-a715-6236-0ed4ac5e4b21[\r][\n]"
   2026-03-03 16:22:07,852 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "amz-sdk-request: attempt=1; max=3[\r][\n]"
   2026-03-03 16:22:07,853 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Authorization: AWS4-HMAC-SHA256 
Credential=A...7/./eu-west-2/s3/aws4_request, 
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;referer;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption;x-amz-server-side-encryption-aws-kms-key-id,
 Signature=5...44[\r][\n]"
   2026-03-03 16:22:07,853 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Content-Type: application/x-directory[\r][\n]"
   2026-03-03 16:22:07,853 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "Referer: 
https://audit.example.org/hadoop/1/op_delete/4022a50f-dd44-4915-b6ec-85c6cb7337c5-0025/?op=op_delete&p1=s3a://stevel-london/f/dir-2cbe3df1-1998-49da-b466-947aae0920f1&pr=stevel&ps=0b35c845-dbf9-4a18-9edd-61469b1f3644&ks=1&cm=StoreDiag&id=4022a50f-dd44-4915-b6ec-85c6cb7337c5-0025&t0=1&fs=4022a50f-dd44-4915-b6ec-85c6cb7337c5&t1=1&ts=1772554927525[\r][\n]";
   2026-03-03 16:22:07,853 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "User-Agent: Hadoop 3.4.3 aws-sdk-java/2.35.4 md/io#sync 
md/http#Apache ua/2.1 api/S3#2.35.x os/Mac_OS_X#26.3 lang/java#17.0.17 
md/OpenJDK_64-Bit_Server_VM#17.0.17+10-LTS md/vendor#Amazon.com_Inc. md/en_GB 
md/rb#u m/F,G hll/cross-region[\r][\n]"
   2026-03-03 16:22:07,853 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]"
   2026-03-03 16:22:07,853 [main] DEBUG http.wire (Wire.java:wire(73)) - 
http-outgoing-2 >> "X-Amz-Date: 20260303T162207Z[\r][\n]"
   2026-03-03 16:22:07,853 [main] DEBUG http.wire (Wire.java:wire(7

Re: [PR] WiP: AWS: S3SignerServlet should strip out more request headers for caching [iceberg]

2026-03-02 Thread via GitHub 


steveloughran commented on PR #15428:
URL: https://github.com/apache/iceberg/pull/15428#issuecomment-398657

   Thinking about how best to do this
   
   * cached request should only include the signed values; the rest jitter 
between irrelevant (referrer) to dangerous (if-modified)
   * the list of headers *known* to be signed can be fixed, (current PR), but 
that's too brittle.
   * Best to parse the signature, get the list of signed headers from there, 
and discard the rest.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]