George Pongracz created SPARK-21702:
---------------------------------------

             Summary: Structured Streaming S3A SSE Encryption Not Applied when 
PartitionBy Used
                 Key: SPARK-21702
                 URL: https://issues.apache.org/jira/browse/SPARK-21702
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.2.0
         Environment: Hadoop 2.7.3: AWS SDK 1.7.4
Hadoop 2.8.1: AWS SDK 1.10.6

            Reporter: George Pongracz
            Priority: Minor


Settings:
      .config("spark.hadoop.fs.s3a.impl", 
"org.apache.hadoop.fs.s3a.S3AFileSystem")
      .config("spark.hadoop.fs.s3a.server-side-encryption-algorithm", "AES256")

When writing to an S3 sink from structured streaming the files are being 
encrypted using AES-256

When introducing a "PartitionBy" the output data files are unencrypted. 

All other supporting files, metadata are encrypted

Suspect write to temp is encrypted and move/rename is not applying the SSE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to