[jira] [Commented] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16390171#comment-16390171 ] Anis Elleuch commented on HADOOP-15267: --- Thanks for merging this patch [~ste...@apache.org]. > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0, 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Fix For: 3.1.0, 3.0.2 > > Attachments: HADOOP-15267-001.patch, HADOOP-15267-002.patch, > HADOOP-15267-003.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387755#comment-16387755 ] Anis Elleuch commented on HADOOP-15267: --- Sure [~ste...@apache.org], thanks for fixing the missing things. Just ping me for whatever you need. > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: HADOOP-15267-001.patch, HADOOP-15267-002.patch, > HADOOP-15267-003.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386962#comment-16386962 ] Anis Elleuch edited comment on HADOOP-15267 at 3/5/18 11:34 PM: [~ste...@apache.org], I added a new patch HDOPP-15267-002.patch which contains the integration test. I tested with a AWS S3 bucket (vadmeste-hadoop, us-east-1) using the following command: {{mvn test -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dtest=ITestS3AHugeFilesSSECDiskBlocks}} Please take a look. was (Author: vadmeste): [~ste...@apache.org] I added a new patch HDOPP-15267-002.patch which contains the integration test. I tested with a AWS S3 bucket (vadmeste-hadoop, us-east-1) using the following command: {{mvn test -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dtest=ITestS3AHugeFilesSSECDiskBlocks}} Please take a look. > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: HADOOP-15267-001.patch, HADOOP-15267-002.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386962#comment-16386962 ] Anis Elleuch commented on HADOOP-15267: --- [~ste...@apache.org] I added a new patch HDOPP-15267-002.patch which contains the integration test. I tested with a AWS S3 bucket (vadmeste-hadoop, us-east-1) using the following command: {{mvn test -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dtest=ITestS3AHugeFilesSSECDiskBlocks}} Please take a look. > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: HADOOP-15267-001.patch, HADOOP-15267-002.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anis Elleuch updated HADOOP-15267: -- Attachment: HADOOP-15267-002.patch > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: HADOOP-15267-001.patch, HADOOP-15267-002.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386201#comment-16386201 ] Anis Elleuch commented on HADOOP-15267: --- [~ste...@apache.org]: I updated the patch with the correct name and made changes you requested (I hope all of them). It doesn't include the integration tests yet but I just wanted to do a review for this progress first. I ran hadoop aws tests ({{cd hadoop-tools/hadoop-aws; mvn test}}) using my AWS S3 bucket vadmeste-hadoop in region us-east-1. > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: HADOOP-15267-001.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anis Elleuch updated HADOOP-15267: -- Attachment: (was: hadoop-fix.patch) > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: HADOOP-15267-001.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anis Elleuch updated HADOOP-15267: -- Attachment: HADOOP-15267-001.patch > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: HADOOP-15267-001.patch, hadoop-fix.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15267) S3A multipart upload fails when SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386049#comment-16386049 ] Anis Elleuch commented on HADOOP-15267: --- Thanks [~ste...@apache.org], I am going to do that. Meanwhile, it looks like branch-3.1 currently generates compiling errors when running the tests.. I am going to work against master to go faster and then I'll see what would be the next steps. > S3A multipart upload fails when SSE-C encryption is enabled > --- > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Assignee: Anis Elleuch >Priority: Critical > Attachments: hadoop-fix.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15267) S3A fails to store my data when multipart size is set ot 5 Mb and SSE-C encryption is enabled
[ https://issues.apache.org/jira/browse/HADOOP-15267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anis Elleuch updated HADOOP-15267: -- Description: When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size to 5 Mb, storing data in AWS doesn't work anymore. For example, running the following code: {code} >>> df1 = spark.read.json('/home/user/people.json') >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") {code} shows the following exception: {code:java} com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload initiate requested encryption. Subsequent part requests must include the appropriate encryption parameters. {code} After some investigation, I discovered that hadoop-aws doesn't send SSE-C headers in Put Object Part as stated in AWS specification: [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] {code:java} If you requested server-side encryption using a customer-provided encryption key in your initiate multipart upload request, you must provide identical encryption information in each part upload using the following headers. {code} You can find a patch attached to this issue for a better clarification of the problem. was: With Spark with Hadoop 3.1.0, when I enable SSE-C encryption and set fs.s3a.multipart.size to 5 Mb, storing data in AWS won't work anymore. For example, running the following code: {code} >>> df1 = spark.read.json('/home/user/people.json') >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") {code} shows the following exception: {code:java} com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload initiate requested encryption. Subsequent part requests must include the appropriate encryption parameters. {code} After some investigation, I discovered that hadoop-aws doesn't send SSE-C headers in Put Object Part as stated in AWS specification: [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] {code:java} If you requested server-side encryption using a customer-provided encryption key in your initiate multipart upload request, you must provide identical encryption information in each part upload using the following headers. {code} You can find a patch attached to this issue for a better clarification of the problem. > S3A fails to store my data when multipart size is set ot 5 Mb and SSE-C > encryption is enabled > - > > Key: HADOOP-15267 > URL: https://issues.apache.org/jira/browse/HADOOP-15267 > Project: Hadoop Common > Issue Type: Bug > Components: fs/s3 >Affects Versions: 3.1.0 > Environment: Hadoop 3.1 Snapshot >Reporter: Anis Elleuch >Priority: Critical > Attachments: hadoop-fix.patch > > > When I enable SSE-C encryption in Hadoop 3.1 and set fs.s3a.multipart.size > to 5 Mb, storing data in AWS doesn't work anymore. For example, running the > following code: > {code} > >>> df1 = spark.read.json('/home/user/people.json') > >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") > {code} > shows the following exception: > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload > initiate requested encryption. Subsequent part requests must include the > appropriate encryption parameters. > {code} > After some investigation, I discovered that hadoop-aws doesn't send SSE-C > headers in Put Object Part as stated in AWS specification: > [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] > {code:java} > If you requested server-side encryption using a customer-provided encryption > key in your initiate multipart upload request, you must provide identical > encryption information in each part upload using the following headers. > {code} > > You can find a patch attached to this issue for a better clarification of the > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15267) S3A fails to store my data when multipart size is set ot 5 Mb and SSE-C encryption is enabled
Anis Elleuch created HADOOP-15267: - Summary: S3A fails to store my data when multipart size is set ot 5 Mb and SSE-C encryption is enabled Key: HADOOP-15267 URL: https://issues.apache.org/jira/browse/HADOOP-15267 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 3.1.0 Environment: Hadoop 3.1 Snapshot Reporter: Anis Elleuch Attachments: hadoop-fix.patch With Spark with Hadoop 3.1.0, when I enable SSE-C encryption and set fs.s3a.multipart.size to 5 Mb, storing data in AWS won't work anymore. For example, running the following code: {code} >>> df1 = spark.read.json('/home/user/people.json') >>> df1.write.mode("overwrite").json("s3a://testbucket/people.json") {code} shows the following exception: {code:java} com.amazonaws.services.s3.model.AmazonS3Exception: The multipart upload initiate requested encryption. Subsequent part requests must include the appropriate encryption parameters. {code} After some investigation, I discovered that hadoop-aws doesn't send SSE-C headers in Put Object Part as stated in AWS specification: [https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html] {code:java} If you requested server-side encryption using a customer-provided encryption key in your initiate multipart upload request, you must provide identical encryption information in each part upload using the following headers. {code} You can find a patch attached to this issue for a better clarification of the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org