[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16343975#comment-16343975 ] Steve Moist commented on HADOOP-15006: -- >what's your proposal for letting the client encryption be an optional feature, >with key? Config If s3a.client.encryption.enabled=true then check for BEZ if exists encrypt objects, else no encryption for the bucket. Or if the BEZI provider is configured as well rather than just the flag. >Is the file length as returned in listings 100% consistent with the amount of >data you get to read? Yes. >I'm not going to touch this right now as its at the too raw stage That's why I submitted it, for you and everyone else to play with to evaluate if this is something that we should move forward with. If needed I can go fix the broken S3Guard/Committer/byte comparison tests and have yetus pass it, but the actual code is going to be about the same. > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf, s3-cse-poc.patch > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341776#comment-16341776 ] Steve Moist commented on HADOOP-15006: -- Ok fixed it. The workflow with block output was causing it to write to disk encrypted and then when it sent it to S3 it encrypted it again causing it to decrypt. So there's a small issue with that in some cases. However, now encryption should work fine for most things. It uses a fixed IV and key to do the encryption, so any files written to S3 will be automatically encrypted/decrypted, so we get some free coverage from the unit tests. It's a quick and dirty prototype so many of the unit tests fail as its not covering all scenarios. I'm able to upload/download files to S3 using the command line without issue. When I view the object in S3 gui, it shows up encrypted, but will automatically decrypt when i do a hdfs get from the cli. Play around with it and let me know what you think. The CryptoStreams work fine, but the integration to fully flesh this out into a feature is what we need to really look at. > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf, s3-cse-poc.patch > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-15006: - Attachment: s3-cse-poc.patch > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf, s3-cse-poc.patch > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339855#comment-16339855 ] Steve Moist edited comment on HADOOP-15006 at 1/25/18 11:19 PM: Found an issue with the poc and am pulling it for now. was (Author: moist): I've attached a quick and dirty proof of concept. It's now using CryptoFSDataInput/OutputStream to write encrypted data to S3. It uses a constant key and iv. I've mainly just ran the entire unit/integration tests against it. I get a bunch of failures as they compare data byte for byte, and as expected they would fail. But the rest of the suite passes with encryption enabled. Play around with it and let me know what you all think. > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-15006: - Attachment: (was: s3-cse-poc.patch) > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339855#comment-16339855 ] Steve Moist commented on HADOOP-15006: -- I've attached a quick and dirty proof of concept. It's now using CryptoFSDataInput/OutputStream to write encrypted data to S3. It uses a constant key and iv. I've mainly just ran the entire unit/integration tests against it. I get a bunch of failures as they compare data byte for byte, and as expected they would fail. But the rest of the suite passes with encryption enabled. Play around with it and let me know what you all think. > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf, s3-cse-poc.patch > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-15006: - Attachment: s3-cse-poc.patch > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf, s3-cse-poc.patch > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[
https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320663#comment-16320663
]
Steve Moist commented on HADOOP-15006:
--
{quote}
you are going to have to handle the fact that its only on the final post where
things are manifest later, with a separate process/host doing the final POST.
that's when any info would have to be persisted to DDB
{quote}
That seems reasonable. I had thought to store the OEMI first and have it act
as a the lock to prevent multiple uploads from confusing who's OEMI is theirs.
By inserting it first, it's hard to tell whether or not the object is still
being uploaded or the host/process/jvm/etc has crashed and abandoned the
upload. There's more to discuss than I originally had thought.
{quote}
One thing to consider is that posix seek() lets you do a relative-to-EOF seek.
Could we offer that, or at least an API call to get the length of a data
source, in our input stream. And then modify the core file formats (Sequence,
ORC, Parquet...) to use the length returned in the open call, rather than a
previously cached value?
{quote}
Can you give me a sample scenario to help visualize what your trying to solve
by this.
> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---
>
> Key: HADOOP-15006
> URL: https://issues.apache.org/jira/browse/HADOOP-15006
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3, kms
>Reporter: Steve Moist
>Priority: Minor
> Attachments: S3-CSE Proposal.pdf
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a
> way that it can leverage HDFS transparent encryption, use the Hadoop KMS to
> manage keys, use the `hdfs crypto` command line tools to manage encryption
> zones in the cloud, and enable distcp to copy from HDFS to S3 (and
> vice-versa) with data still encrypted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[
https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317345#comment-16317345
]
Steve Moist commented on HADOOP-15006:
--
{quote}
Do have any good links for further reading on the crypto algorithms,
particularly the NoPadding variant you mention? (How do lengths and byte
offsets map from the user data to the encrypted stream?)
{quote}
I've got a few links about general block ciphers and padding. I'll post more
as I find them later.
* http://web.cs.ucdavis.edu/~rogaway/papers/modes.pdf is a good(and lengthy)
doc on encryption, look at page 5 for a summary and then page 45 for more on
CTR.
* https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Padding
Obligatory wikipedia page
* https://www.cryptosys.net/pki/manpki/pki_paddingschemes.html
{quote}
(How do lengths and byte offsets map from the user data to the encrypted
stream?)
{quote}
They should map 1-1. AES works on a fixed block size of 16 bytes. So you
would read from 0->15 bytes, 16->31, etc. It means you can't read 3->18
directly, you'd have to read 0->31 to read 3->18. Not sure exactly but I would
imagine HDFS transparent encryption also has the same issue that has already
been solved. It would just mean we would have to get the before block as well
to properly decrypt. CTR allows for random encryption/decryption so I don't
expect this to be a problem performing encryption/decryption . It's just a
minor technical point. So far in my testing I haven't hit it, but I also
haven't been directly invoking the MultiPartUpload. This is the only issue I
see when randomly reading/writing blocks and it's easily solvable.
{quote}
What are the actual atomicity requirements?
{quote}
This is a good question. The main atomicity requirements that I have is that
once the S3a stream is closed and the object committed that the OEMI is also
committed. I haven't fully worked around that from a specific code perspective
yet.
{quote}
Specifically, how do we handle multiple clients racing to create the same path?
{quote}
Using OEMI Storage Option #5: Suppose userA uploads objectA, with OEMIA to key1
and userB uploads objectB with OEMIB to key1. S3 is doesn't guarantee which
one is the winner, so it is possible that objectA is stored with OEMIB. This
shouldn't happen if OEMI is stored as object metadata. It could be done such
that we create a "lock" on the DynamoDB row that userA owns that location
preventing the upload of objectB. In HDFS, once the INode is created, it
should prevent userB from creating that file, perhaps we should do the same for
S3?
{quote}
Also since the scope of the encryption zone is the bucket, we could get by with
a very low provisioned I/O budget on the Dynamo table and save money, no?
{quote}
Yea we should be able to, I believe the only requirements is that the table can
have things inserted and read. IIRC, each bucket gets their own S3a jvm (or
somethign to that effect) so at least at startup we can cache its EZ
information.
> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---
>
> Key: HADOOP-15006
> URL: https://issues.apache.org/jira/browse/HADOOP-15006
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3, kms
>Reporter: Steve Moist
>Priority: Minor
> Attachments: S3-CSE Proposal.pdf
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a
> way that it can leverage HDFS transparent encryption, use the Hadoop KMS to
> manage keys, use the `hdfs crypto` command line tools to manage encryption
> zones in the cloud, and enable distcp to copy from HDFS to S3 (and
> vice-versa) with data still encrypted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[
https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310101#comment-16310101
]
Steve Moist edited comment on HADOOP-15006 at 1/3/18 7:02 PM:
--
{quote}
Before worrying about these, why not conduct some experiments? You could take
S3A and modify it to always encrypt client side with the same key, then run as
many integration tests as you can against it (Hive, Spark, impala, ...), and
see what fails. I think that should be a first step to anything client-side
related
{quote}
I wrote a simple proof of concept back in May using the HDFS Crytro Streams
wrapping the S3 streams with a fixed AES key and IV. I was able to run the S3
integration tests without issue, terragen/sort/verify without issue and write
various files (of differing sizes) and compare the check sums. It's given me
enough confidence to move forward back then with writing the original proposal.
Unfortunately, I've seemed to misplaced the work since it's been so long.
I'll work on re-creating it in the next few weeks and post it here; I've got a
deadline I've got to focus on for now instead. So after that I'll run some
Hive/Impala/etc integration tests. Besides AES/CTR/NoPadding should generate a
cipher text the same size as the plain text unlike the AWS's SDK's
AES/CBC/PKCS5Padding which is causing the file size to change.
was (Author: moist):
{quote}
Before worrying about these, why not conduct some experiments? You could take
S3A and modify it to always encrypt client side with the same key, then run as
many integration tests as you can against it (Hive, Spark, impala, ...), and
see what fails. I think that should be a first step to anything client-side
related
{quote}
I wrote a simple proof of concept back in May using the HDFS Crytro Streams
wrapping the S3 streams with a fixed AES key and IV. I was able to run the S3
integration tests without issue, terragen/sort/verify without issue and write
various files (of differing sizes) and compare the check sums. It's given me
enough confidence to move forward back then with writing the original proposal.
Unfortunately, I've seemed to misplaced the work since it's been so long.
I'll work on re-creating it in the next few weeks and post it here; I've got a
deadline I've got to focus on for now instead. Besides AES/CTR/NoPadding
should generate a cipher text the same size as the plain text unlike the AWS's
SDK's AES/CBC/PKCS5Padding which is causing the file size to change.
> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---
>
> Key: HADOOP-15006
> URL: https://issues.apache.org/jira/browse/HADOOP-15006
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3, kms
>Reporter: Steve Moist
>Priority: Minor
> Attachments: S3-CSE Proposal.pdf
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a
> way that it can leverage HDFS transparent encryption, use the Hadoop KMS to
> manage keys, use the `hdfs crypto` command line tools to manage encryption
> zones in the cloud, and enable distcp to copy from HDFS to S3 (and
> vice-versa) with data still encrypted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[
https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310101#comment-16310101
]
Steve Moist commented on HADOOP-15006:
--
{quote}
Before worrying about these, why not conduct some experiments? You could take
S3A and modify it to always encrypt client side with the same key, then run as
many integration tests as you can against it (Hive, Spark, impala, ...), and
see what fails. I think that should be a first step to anything client-side
related
{quote}
I wrote a simple proof of concept back in May using the HDFS Crytro Streams
wrapping the S3 streams with a fixed AES key and IV. I was able to run the S3
integration tests without issue, terragen/sort/verify without issue and write
various files (of differing sizes) and compare the check sums. It's given me
enough confidence to move forward back then with writing the original proposal.
Unfortunately, I've seemed to misplaced the work since it's been so long.
I'll work on re-creating it in the next few weeks and post it here; I've got a
deadline I've got to focus on for now instead. Besides AES/CTR/NoPadding
should generate a cipher text the same size as the plain text unlike the AWS's
SDK's AES/CBC/PKCS5Padding which is causing the file size to change.
> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---
>
> Key: HADOOP-15006
> URL: https://issues.apache.org/jira/browse/HADOOP-15006
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3, kms
>Reporter: Steve Moist
>Priority: Minor
> Attachments: S3-CSE Proposal.pdf
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a
> way that it can leverage HDFS transparent encryption, use the Hadoop KMS to
> manage keys, use the `hdfs crypto` command line tools to manage encryption
> zones in the cloud, and enable distcp to copy from HDFS to S3 (and
> vice-versa) with data still encrypted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[
https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308621#comment-16308621
]
Steve Moist commented on HADOOP-15006:
--
{quote}
It appears there will be NO CHANGES to the KMS, right?
{quote}
Yes there are no changes to the KMS, and I expect to be able to do all KMS
actions through the existing API calls.
{quote}
Do you intend to add a hadoop crypto wrapper, or is there intention to change
hdfs's CryptoAdmin? I'd suggest the former.
{quote}
I'm planning to change the CryptoAdmin to support S3, but it would call out
into the S3aCryptoAdmin and change the CLI invocation. While this may be a
bigger change now, renaming the command can start paving the way for Azure CSE
way down the road.
{quote}
About metadata persistence, does option #5 mean to add a new column to the S3G
DynamoDB table, to store the edeks along with the other metadata of a file? I
feel this is the safest way because we don't have to worry about consistency
particular to edeks.
{quote}
No, it is not meant to be part of S3Guard as it has the ability to delete the
table and refresh it. Doing so would cause loss of EDEK's and therefore data
loss.
{quote}
How is the EZ information stored?
{quote}
It is stored as BEZI. Simarily, the NN could read the BEZI or S3a could cache
that as well.
{quote}
What's the behavior when KMS is slow and we create a file? If the generateEDEK
exhausted cache, where does it hang? When this happens does it impact other
operation? (This is a pain for NN due to the single namespace lock, may not be
an issue for s3a but I'm curious)
{quote}
S3a would have to wait for the KMS to generate an EDEK. This I'm not sure how
it would affect other operations.
{quote}
I saw raw bytes mentioned. For hdfs, the way to access raw bytes is through a
special path: /.reserved/raw/original_path. How is this done in s3?
{quote}
That piece I'm not too sure yet about. It could be a virtual object key (such
as bucket-name/.reserved/raw/original_path) that isn't actually created in S3
and interpreted on a copy or move command to not decrypt data. I would like to
include it as potentially users could use DistCP to copy encrypted data from an
HDFS EZ to a S3 EZ without having to decrypt it as they share the same EZK. I
think that feature would be a great thing to do. But this subject needs to be
fleshed out more.
> Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
> ---
>
> Key: HADOOP-15006
> URL: https://issues.apache.org/jira/browse/HADOOP-15006
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs/s3, kms
>Reporter: Steve Moist
>Priority: Minor
> Attachments: S3-CSE Proposal.pdf
>
>
> This is for the proposal to introduce Client Side Encryption to S3 in such a
> way that it can leverage HDFS transparent encryption, use the Hadoop KMS to
> manage keys, use the `hdfs crypto` command line tools to manage encryption
> zones in the cloud, and enable distcp to copy from HDFS to S3 (and
> vice-versa) with data still encrypted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288396#comment-16288396 ] Steve Moist commented on HADOOP-15006: -- I don't think anyone's started it. I posted the design doc in hopes of others looking at it and critiquing it in the background while I focus on other things, that once enough people had reviewed it, to start on it then. The changes to the Hadoop CLI, KMS and other components was what worried me about it. It's bigger in scope than just S3a. In the proposal I made, we didn't have an issue with the cipher text length and plaintext length as we used CTR with no padding vs the CBC with PKCS5Padding that the AWS sdk uses. I wrote a quick prototype using AES/CTR/NoPadding and ran all the integration tests against it and it ran without issue and did diffs on the before/after of upload/download along with TerraSort and had no issues. > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286574#comment-16286574 ] Steve Moist commented on HADOOP-15006: -- Hey [~fabbri] and [~steve_l], any traction on this? > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286572#comment-16286572 ] Steve Moist commented on HADOOP-14209: -- Is this still something that is needed to be done? > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Trivial > Labels: newbie > Attachments: HADOOP-14209-001.patch, HADOOP-14209-branch-2-001.patch > > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Encrypt S3A data client-side with AWS SDK
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234365#comment-16234365 ] Steve Moist commented on HADOOP-13887: -- >I can see the appeal of some form of support for this purely for some >backup/restore process, I agree, that's a scenario I am going to cover in the other proposal. >People will end up encrypting their data, then be filing bugs/support calls >trying to understand why their queries are all failing. Oh yes they will. >It also isn't going to interact with any other S3 client, which is a >significant limitation The aws S3 cse sdk also has that limitation. IIRC it is also written in Java which makes portability a concern. At least with the Hadoop KMS, it exposes REST endpoints to encrypt/decrypt keys making it more platform independent. So while utitlities don't integrate currently with it, it doesn't prevent them from in the future from doing so. Even a lot of the AWS services don't integrate with the cse sdk. I created HADOOP-15006 and renamed this jira. I will let [~Igor Mazur] or [~steve_l] close the ticket as I am unsure of how to do so. > Encrypt S3A data client-side with AWS SDK > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-15006: - Attachment: S3-CSE Proposal.pdf > Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS > --- > > Key: HADOOP-15006 > URL: https://issues.apache.org/jira/browse/HADOOP-15006 > Project: Hadoop Common > Issue Type: New Feature > Components: fs/s3, kms >Reporter: Steve Moist >Priority: Minor > Attachments: S3-CSE Proposal.pdf > > > This is for the proposal to introduce Client Side Encryption to S3 in such a > way that it can leverage HDFS transparent encryption, use the Hadoop KMS to > manage keys, use the `hdfs crypto` command line tools to manage encryption > zones in the cloud, and enable distcp to copy from HDFS to S3 (and > vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
Steve Moist created HADOOP-15006: Summary: Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS Key: HADOOP-15006 URL: https://issues.apache.org/jira/browse/HADOOP-15006 Project: Hadoop Common Issue Type: New Feature Components: fs/s3, kms Reporter: Steve Moist Priority: Minor This is for the proposal to introduce Client Side Encryption to S3 in such a way that it can leverage HDFS transparent encryption, use the Hadoop KMS to manage keys, use the `hdfs crypto` command line tools to manage encryption zones in the cloud, and enable distcp to copy from HDFS to S3 (and vice-versa) with data still encrypted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13887) Encrypt S3A data client-side with AWS SDK
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-13887: - Summary: Encrypt S3A data client-side with AWS SDK (was: Support for client-side encryption in S3A file system) > Encrypt S3A data client-side with AWS SDK > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219616#comment-16219616 ] Steve Moist commented on HADOOP-13887: -- [~steve_l] Should we continue discussion on this or table it for now? > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109860#comment-16109860 ] Steve Moist edited comment on HADOOP-13887 at 8/1/17 10:00 PM: --- [~steve_l] I do like object tags to store the OEMI on, but the thing that worries about it for me is users modifying the tags. I don't think there is an IAM policy to prevent users from modifying say a specific Hadoop tag unless they are an admin. I worry about accidental data loss by either a user accidentally deleting/editing the EDEK or a rogue actor deleting tags. We would still need a place to store the BEZI, but we could just use a tag on the bucket. I'll have to take a look into HADOOP-13786. If you do have time, do take a look deeper into it. I wanted to get the idea out there and start getting feedback. I'm certain I'll have many revisions to do. was (Author: moist): [~steve_l] I do like object tags to store the OEMI on, but the thing that worries about it for me is users modifying the tags. I don't think there is an IAM policy to prevent users from modifying say a Hadoop tag unless they are an admin. I worry about accidental data loss by either a user accidentally deleting/editing the EDEK or a rogue actor deleting tags. We would still need a place to store the BEZI, but we could just use a tag on the bucket. I'll have to take a look into HADOOP-13786. If you do have time, do take a look deeper into it. I wanted to get the idea out there and start getting feedback. I'm certain I'll have many revisions to do. > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109860#comment-16109860 ] Steve Moist commented on HADOOP-13887: -- [~steve_l] I do like object tags to store the OEMI on, but the thing that worries about it for me is users modifying the tags. I don't think there is an IAM policy to prevent users from modifying say a Hadoop tag unless they are an admin. I worry about accidental data loss by either a user accidentally deleting/editing the EDEK or a rogue actor deleting tags. We would still need a place to store the BEZI, but we could just use a tag on the bucket. I'll have to take a look into HADOOP-13786. If you do have time, do take a look deeper into it. I wanted to get the idea out there and start getting feedback. I'm certain I'll have many revisions to do. > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096761#comment-16096761 ] Steve Moist commented on HADOOP-13887: -- I think they can easily co-exist as distinct options. But I haven't really considered a migration path or supporting both simultaneously or even intermixing them. > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096558#comment-16096558 ] Steve Moist commented on HADOOP-13887: -- Hey [~steve_l] and [~Igor Mazur], I've spent some time outlining a different approach that I've been working on. Please take a look at the attached "S3-CSE Proposal.pdf" and let me know what you all think. I think this approach would be better overall for the Hadoop ecosystem and gladly welcome feedback. > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-13887: - Attachment: S3-CSE Proposal.pdf > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch, S3-CSE Proposal.pdf > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[ https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030185#comment-16030185 ] Steve Moist commented on HADOOP-13887: -- So I've been taking a look at a different approach to S3 Client Side Encryption and prototyping it. Still working through some of the feasibility and impact to the ecosystem right now. I'd like to put it out here and see if it's something upstream would like to adopt. > Support for client-side encryption in S3A file system > - > > Key: HADOOP-13887 > URL: https://issues.apache.org/jira/browse/HADOOP-13887 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Jeeyoung Kim >Assignee: Igor Mazur >Priority: Minor > Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch, > HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch, > HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch, > HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch, > HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch, > HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch, > HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch > > > Expose the client-side encryption option documented in Amazon S3 > documentation - > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html > Currently this is not exposed in Hadoop but it is exposed as an option in AWS > Java SDK, which Hadoop currently includes. It should be trivial to propagate > this as a parameter passed to the S3client used in S3AFileSystem.java -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993019#comment-15993019 ] Steve Moist commented on HADOOP-14209: -- Fixed all the formatting issues in trunk to be in accordance with checkstyle. > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Trivial > Labels: newbie > Attachments: HADOOP-14209-001.patch, HADOOP-14209-branch-2-001.patch > > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989151#comment-15989151 ] Steve Moist commented on HADOOP-14209: -- Minor formatting change in trunk for the validation tests. Had to fix one of the tests as there was an outdated error message. > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Trivial > Labels: newbie > Attachments: HADOOP-14209-001.patch, HADOOP-14209-branch-2-001.patch > > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-14209: - Status: Patch Available (was: In Progress) > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 3.0.0-alpha2, 2.9.0 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Trivial > Labels: newbie > Attachments: HADOOP-14209-001.patch, HADOOP-14209-branch-2-001.patch > > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-14209: - Attachment: HADOOP-14209-branch-2-001.patch HADOOP-14209-001.patch Tested against aws us-west-2. > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Trivial > Labels: newbie > Attachments: HADOOP-14209-001.patch, HADOOP-14209-branch-2-001.patch > > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist reassigned HADOOP-14209: Assignee: Steve Moist (was: Sean Mackrory) > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Trivial > Labels: newbie > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-14209 started by Steve Moist. > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Steve Moist >Assignee: Steve Moist >Priority: Trivial > Labels: newbie > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13887) Support for client-side encryption in S3A file system
[
https://issues.apache.org/jira/browse/HADOOP-13887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983638#comment-15983638
]
Steve Moist commented on HADOOP-13887:
--
If you don't mind. I'd like to jump in with some thoughts.
1). Rename S3AClientEncryptionMethods.KMS to
S3AClientEncryptionMethods.AWS-KMS. Since Hadoop already has a KMS, it might
be confusing for users that think that this might be the Hadoop KMS instead of
the AWS-KMS.
2). Refactor S3AEncryptionMethods to S3AServerEncryptionMethods for clarity
and consistancy with S3AClientEncryptionMethods.
3). It looks to me if S3ClientFactory.getAmazonS3EncryptionClient is configured
with S3AClientEncryptionMethods.NONE, it will try to load
custom encryption materials and throw a IllegalArgumentException.
{quote}
the stack traces should go into the troubleshooting section in index.md, or
maybe we could add a whole new page on encryption?
{quote}
Makes sense to me to create a new page for encryption, since I just added more
troubleshooting for SSE.
{quote}
most (all?) of us don't know about how s3 client side encryption works, so
these details are not something we necessarily have valid opinions on.
{quote}
I've actually worked with the Java api before with S3 CSE. So I can help with
reviewing.
> Support for client-side encryption in S3A file system
> -
>
> Key: HADOOP-13887
> URL: https://issues.apache.org/jira/browse/HADOOP-13887
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Jeeyoung Kim
>Assignee: Igor Mazur
>Priority: Minor
> Attachments: HADOOP-13887-002.patch, HADOOP-13887-007.patch,
> HADOOP-13887-branch-2-003.patch, HADOOP-13897-branch-2-004.patch,
> HADOOP-13897-branch-2-005.patch, HADOOP-13897-branch-2-006.patch,
> HADOOP-13897-branch-2-008.patch, HADOOP-13897-branch-2-009.patch,
> HADOOP-13897-branch-2-010.patch, HADOOP-13897-branch-2-012.patch,
> HADOOP-13897-branch-2-014.patch, HADOOP-13897-trunk-011.patch,
> HADOOP-13897-trunk-013.patch, HADOOP-14171-001.patch
>
>
> Expose the client-side encryption option documented in Amazon S3
> documentation -
> http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html
> Currently this is not exposed in Hadoop but it is exposed as an option in AWS
> Java SDK, which Hadoop currently includes. It should be trivial to propagate
> this as a parameter passed to the S3client used in S3AFileSystem.java
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-14305: - Attachment: HADOOP-14305-branch-2-002.patch > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch, > HADOOP-14305-branch-2-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981459#comment-15981459 ] Steve Moist commented on HADOOP-14305: -- us-west-2 (Oregon) > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-14305: - Status: Patch Available (was: In Progress) > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980160#comment-15980160 ] Steve Moist commented on HADOOP-14305: -- Uploaded a patch for trunk, if everyone is satisfied with it I will then work on back porting it to branch-2. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-14305: - Attachment: HADOOP-14305-002.patch > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch, HADOOP-14305-002.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973589#comment-15973589 ] Steve Moist commented on HADOOP-14305: -- I've made the pom changes, but want to spend some time writing some more unit tests that deal with SSE-C. I'll submit a patch in a few days. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973319#comment-15973319 ] Steve Moist commented on HADOOP-14305: -- Sounds good. I'll submit a few patches today. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist reassigned HADOOP-14305: Assignee: Steve Moist > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work started] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-14305 started by Steve Moist. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Assignee: Steve Moist >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[ https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971778#comment-15971778 ] Steve Moist commented on HADOOP-14305: -- I'm pretty sure it's SSE-C related. I disabled the SSE-C tests and it ran fine in parallel mode. So I entirely suspect its that its using the wrong key, wrong method or no key supplied when trying to re-access the fork-#/ prefix in the tests (either with SSE-C, and the other tests failing because it's not cleaned up properly) and s3 is failing due to the wrong key being provided. If I do a "aws s3api head-object --bucket my-bucket --key sse-c-encrypted.txt", I get a "An error occurred (400) when calling the HeadObject operation: Bad Request" when trying to get the head of an object with no encryption key, or the wrong encryption key. Which is the same http error in the failed tests. SSE-C was one of those s3 features where the encryption key must always be provided for all operations or you get a 400 error. So I suspect this is the real culprit of the issue. In general, I don't really advise people to use SSE-C. In the web ui, you can't tell if it's encrypted, which encryption key was used, nor can you get the meta-data about the object without the correct encryption key through the command line. If a customer wishes to use this, the best solution is to use 1 SSE-C key across the cluster. If they wish to use multiple, they need to keep track of which key maps to where and should NEVER share buckets. I also recommend the documentation advising only to use this for advanced users. > S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus > --- > > Key: HADOOP-14305 > URL: https://issues.apache.org/jira/browse/HADOOP-14305 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3, test >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > Attachments: HADOOP-14305-001.patch > > > The S3a encryption tests all run in parallel (they were interfering with each > other, apparently). This adds ~1 min to the test runs. > They should run in serial. That they fail when this is attempted due to Bad > Auth problems must be considered a serious problem, as is indicates issues > related to working with SSE encryption from hadoop -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14305) S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
[
https://issues.apache.org/jira/browse/HADOOP-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969466#comment-15969466
]
Steve Moist commented on HADOOP-14305:
--
{quote}
Steve Moist : have you been using this feature yet? In particular, using it
with writers to the same bucket with different policies? It may be that you
just can't do this —something the release notes will need to cover.
{quote}
I've mainly been using it during testing with a cluster and parts of a demo.
Most of it has been fairly light usage, except when I ran terragen a few times.
I haven't seen this kind of issue. During a demo I did change from SSE-S3 to
SSE-KMS to SSE-C to write to the same bucket, but I used Cloudera Manager to
push those changes to the cluster and restarted the services. So I never had
multiple threads running with different encryption types.
> S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
> ---
>
> Key: HADOOP-14305
> URL: https://issues.apache.org/jira/browse/HADOOP-14305
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 2.9.0
>Reporter: Steve Loughran
>Priority: Minor
> Attachments: HADOOP-14305-001.patch
>
>
> The S3a encryption tests all run in parallel (they were interfering with each
> other, apparently). This adds ~1 min to the test runs.
> They should run in serial. That they fail when this is attempted due to Bad
> Auth problems must be considered a serious problem, as is indicates issues
> related to working with SSE encryption from hadoop
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967987#comment-15967987 ] Steve Moist commented on HADOOP-13075: -- If I remember correctly, I kept getting the runs overwriting each other in S3 when run in parallel. This was mainly due to them being subclasses. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch, > HADOOP-13075-003.patch, HADOOP-13075-branch2.002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-14209) Remove @Ignore from valid S3a test.
Steve Moist created HADOOP-14209: Summary: Remove @Ignore from valid S3a test. Key: HADOOP-14209 URL: https://issues.apache.org/jira/browse/HADOOP-14209 Project: Hadoop Common Issue Type: Test Components: fs/s3 Affects Versions: 3.0.0-alpha2 Reporter: Steve Moist Priority: Trivial The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is ignored through the @Ignore annotation, this should be removed as it is a valid test class. This was a minor mistake introduced during development of HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14209) Remove @Ignore from valid S3a test.
[ https://issues.apache.org/jira/browse/HADOOP-14209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935012#comment-15935012 ] Steve Moist commented on HADOOP-14209: -- This is in trunk and branch-2 and should be fixed in both places. > Remove @Ignore from valid S3a test. > --- > > Key: HADOOP-14209 > URL: https://issues.apache.org/jira/browse/HADOOP-14209 > Project: Hadoop Common > Issue Type: Test > Components: fs/s3 >Affects Versions: 3.0.0-alpha2 >Reporter: Steve Moist >Priority: Trivial > > The class org.apache.hadoop.fs.s3a.ITestS3AEncryptionAlgorithmValidation is > ignored through the @Ignore annotation, this should be removed as it is a > valid test class. This was a minor mistake introduced during development of > HADOOP-13075. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870245#comment-15870245 ] Steve Moist commented on HADOOP-13075: -- Is that something that we need to fix? > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch, > HADOOP-13075-003.patch, HADOOP-13075-branch2.002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-13075: - Attachment: HADOOP-13075-branch2.002.patch HADOOP-13075-003.patch Branch-2 patch and trunk patches with updated ignoring KMS test if no key is defined. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch, > HADOOP-13075-003.patch, HADOOP-13075-branch2.002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856794#comment-15856794 ] Steve Moist commented on HADOOP-13075: -- So with SSE-KMS, there's 2 cases, one where the user doesn't submit a KMS key and where they do. I added a new test that will run with the default key(aws auto-configures this) . The ones that require a key still exist, and now if the user doesn't specify said key, they will be skipped. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854759#comment-15854759 ] Steve Moist commented on HADOOP-13075: -- [~steve_l] I've asked around on our side and we don't have anyone who's a committer that has enough bandwidth to run this. Is it all right if we have a non-committer run it and report back? > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-13075: - Attachment: HADOOP-13075-002.patch Corrected patch. The precondition lines are still too long with the static import. The other checkstyle issues were already corrected in the git commit in the fork but for some reason not included in the first patch. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Attachments: HADOOP-13075-001.patch, HADOOP-13075-002.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Moist updated HADOOP-13075: - Attachment: HADOOP-13075-001.patch trunk patch > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Andrew Olson >Assignee: Steve Moist > Attachments: HADOOP-13075-001.patch > > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843591#comment-15843591 ] Steve Moist commented on HADOOP-13075: -- I ran some tests using the minicluster against the default s3 endpoint where buckets were configured in Oregon and encryption keys stored in Oregon. One thing to note using SSE-C, there is nothing in the AWS S3 GUI that shows that the file is encrypted. >we should be able to verify that data written with one key cannot be > parsed if a different fs + key is used to read it. I uploaded FileA using the minicluster under aws:kms key1, then did a fs -mv after restarting the minicluster with aws:kms key2. It successfully moved it to FileA' encrypted with aws:kms key2. If SSE-C is enabled, it will throw an error. > we should see what happens if you try to read unencrypted data with an FS > with encryption enabled I uploaded FileA through the AWS S3 GUI. I moved a file from fileA to fileA' where fileA was unencrypted with SSE-KMS enabled. FileA' became encrypted under the aws kms key configured. If SSE-C is enabled, it will throw an error. > maybe: if a bucket is set up to require encryption, then unencrypted data cannot be written, encrypted can. This implies that the tester will need a special bucket for this test & declare it in the configs. The user can still upload data through the GUI or AWS cli and not have it be encrypted. Based on #2, any new file will be encrypted with SSE-S3/SSE-KMS and any copied/moved file will now be encrypted. This point seems like it will be hard to enforce. I ran the full s3a suite with aws:kms enabled and it ran fine on the defaults s3 enpdoint in the oregon region. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840518#comment-15840518 ] Steve Moist commented on HADOOP-13075: -- Updated with changes. One issue I found with refactoring the encryptionMethodAlgorthm string into an enum is that it breaks the test ITestS3AEncryptionAlgorithmPropagation. In this test, it sets the algorithm to "DES" and when it places a S3 request, S3 then validates that and throws an exception. However this is no longer the case, in S3aFileSystem.initialize, an exception is thrown here when trying to convert that string into an enum and the tests fail during setup. Should we keep this test or remove it as this behavior isn't expected anymore? In general I have found that I cannot run most tests in parallel. As an example, one class expects an empty dir, but the test suites spawn subfolders in the root of the bucket for their tests to be isolated. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840149#comment-15840149 ] Steve Moist commented on HADOOP-13075: -- Ok, that's what I originally thought. I will leave the code as is by introducing the new SERVER_SIDE_ENCRYPTION_KEY. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840118#comment-15840118 ] Steve Moist commented on HADOOP-13075: -- [~ste...@apache.org] So you're saying don't change the name of the constant but changing the contents of the constant is ok. In this case is that something that we want to do? [~lmccay] You make a good point. You can switch the mode from SSE_C to SSS_KMS to print out the contents of the key. This is mainly just for the toString method and can be removed. But if the user has the ability to switch modes, wouldn't they also have access to config with the key? I also don't believe the use case needs key versioning. If that were to be supported that would have much larger implications than this. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839021#comment-15839021 ] Steve Moist commented on HADOOP-13075: -- Updated pull request with [~steve_l]'s comments. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838654#comment-15838654 ] Steve Moist commented on HADOOP-13075: -- [~fabbri] One of Frederico's original changes was removing that. It was commented on saying not to do so (in branch-2). It would be a simpler change but I feel overloading that wouldn't be as clear. I'm not sure what customers are using that field, the only use of it is in ITestS3AEncryption. Since this is 3.0 of Hadoop, is it possible to remove old fields? I'm more supportie of removing that field. > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838352#comment-15838352 ] Steve Moist commented on HADOOP-13075: -- Sounds good, I'll make changes around the Enum and convert the if statements to a switch. > L 912 : What if the user has selected SSE_C and the server key is blank? > That's downgraded to a no-op, now, yes? I'd rather that was treated as a failure: if you ask for server-side encryption and don't supply a key, FS init must fail. I'll move that check into initialize. Is there a preference on what type of exception to be thrown? >Blocker: The server-side encryption key must be supported; use >S3AUtils.getPassword for this. By this do you mean convert instances of request.setSSECustomerKey(new SSECustomerKey(serverSideEncryptionKey)); with request.setSSECustomerKey(new SSECustomerKey(S3AUtils.getPassword(getConf(), Constants.SERVER_SIDE_ENCRYPTION_KEY)); > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838104#comment-15838104 ] Steve Moist commented on HADOOP-13075: -- Hi, I've taken Federico's changes and updated them to be consistent with trunk's changes and submitted a pull request to merge into trunk. I will be working on a separate one for branch-2 > Add support for SSE-KMS and SSE-C in s3a filesystem > --- > > Key: HADOOP-13075 > URL: https://issues.apache.org/jira/browse/HADOOP-13075 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Reporter: Andrew Olson >Assignee: Federico Czerwinski > > S3 provides 3 types of server-side encryption [1], > * SSE-S3 (Amazon S3-Managed Keys) [2] > * SSE-KMS (AWS KMS-Managed Keys) [3] > * SSE-C (Customer-Provided Keys) [4] > Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 > (HADOOP-10568) -- the underlying aws-java-sdk makes that very simple [5]. > With native support in aws-java-sdk already available it should be fairly > straightforward [6],[7] to support the other two types of SSE with some > additional fs.s3a configuration properties. > [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html > [2] > http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html > [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html > [4] > http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html > [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html > [6] > http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java > [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
