George Pongracz created SPARK-21702:

             Summary: Structured Streaming S3A SSE Encryption Not Applied when 
PartitionBy Used
                 Key: SPARK-21702
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.2.0
         Environment: Hadoop 2.7.3: AWS SDK 1.7.4
Hadoop 2.8.1: AWS SDK 1.10.6

            Reporter: George Pongracz
            Priority: Minor

      .config("spark.hadoop.fs.s3a.server-side-encryption-algorithm", "AES256")

When writing to an S3 sink from structured streaming the files are being 
encrypted using AES-256

When introducing a "PartitionBy" the output data files are unencrypted. 

All other supporting files, metadata are encrypted

Suspect write to temp is encrypted and move/rename is not applying the SSE.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to