George Pongracz created SPARK-21702:
---------------------------------------
Summary: Structured Streaming S3A SSE Encryption Not Applied when
PartitionBy Used
Key: SPARK-21702
URL: https://issues.apache.org/jira/browse/SPARK-21702
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 2.2.0
Environment: Hadoop 2.7.3: AWS SDK 1.7.4
Hadoop 2.8.1: AWS SDK 1.10.6
Reporter: George Pongracz
Priority: Minor
Settings:
.config("spark.hadoop.fs.s3a.impl",
"org.apache.hadoop.fs.s3a.S3AFileSystem")
.config("spark.hadoop.fs.s3a.server-side-encryption-algorithm", "AES256")
When writing to an S3 sink from structured streaming the files are being
encrypted using AES-256
When introducing a "PartitionBy" the output data files are unencrypted.
All other supporting files, metadata are encrypted
Suspect write to temp is encrypted and move/rename is not applying the SSE.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
