[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17332958#comment-17332958 ] Akira Ajisaka commented on HADOOP-17255: Since I'm not interested in this issue, removed the target versions. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226947#comment-17226947 ] Daryn Sharp commented on HADOOP-17255: -- When you re-do this, I'd rather not have file context involved when the rest of the implementation is filesystem based. I'll echo earlier statements on security. Storing EZ secret keys in unencrypted hdfs is a horrible idea. Esp. if you are storing them on the same NN that has the EZs protected by the keys, I'm not sure why you are even using EZs... > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[
https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226385#comment-17226385
]
Ahmed Hussein commented on HADOOP-17255:
[~weichiu], [~aajisaka]
I noticed that there are two failures in the [qbt report
build|https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/315/#showFailuresLink]
following the merge of this PR.
Can you take a quick look to see if it is caused by the PR or not?
{code:bash}
org.apache.hadoop.crypto.key.TestKeyProviderFactory.testGetProviderViaURI
Failing for the past 1 build (Since Failed#315 )
Took 0.89 sec.
Error Message
fs.AbstractFileSystem.file.impl=null: No AbstractFileSystem configured for
scheme: file
Stacktrace
org.apache.hadoop.fs.UnsupportedFileSystemException:
fs.AbstractFileSystem.file.impl=null: No AbstractFileSystem configured for
scheme: file
at
org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:176)
at
org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:265)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:342)
at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:339)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at
org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:339)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:465)
at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:491)
at
org.apache.hadoop.crypto.key.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:136)
at
org.apache.hadoop.crypto.key.JavaKeyStoreProvider.(JavaKeyStoreProvider.java:90)
at
org.apache.hadoop.crypto.key.JavaKeyStoreProvider$Factory.createProvider(JavaKeyStoreProvider.java:661)
at
org.apache.hadoop.crypto.key.KeyProviderFactory.get(KeyProviderFactory.java:96)
at
org.apache.hadoop.crypto.key.TestKeyProviderFactory.testGetProviderViaURI(TestKeyProviderFactory.java:429)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{code}
{code:bash}
org.apache.hadoop.crypto.key.TestKeyProviderFactory.testJksProvider
Failing for the past 1 build (Since Failed#315 )
Took 3.4 sec.
Error Message
expected null, but was:
Stacktrace
java.lang.AssertionError: expected null, but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotNull(Assert.java:755)
at org.junit.Assert.assertNull(Assert.java:737)
at org.junit.Assert.assertNull(Assert.java:747)
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[
https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225623#comment-17225623
]
Wei-Chiu Chuang commented on HADOOP-17255:
--
{quote} * Probably there are many users, so the quality of Ranger KMS is better
than that of Hadoop KMS.{quote}
That is dubious. As far as I know, RangerKMS is a fork of HadoopKMS and for the
most part, they simply port fixes/features from HadoopKMS over there. There is
little commits in the ranger kms too.
Therefore, we should still maintain HadoopKMS code.
> JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
> --
>
> Key: HADOOP-17255
> URL: https://issues.apache.org/jira/browse/HADOOP-17255
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws
> FileNotFoundException if the src does not exist. However,
> JavaKeyStoreProvider#renameOrFail calls the old rename API. In
> DistributedFileSystem, the old API returns false if the src does not exist.
> That way JavaKeyStoreProvider fails to create a new key if the keystore is
> HDFS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17201920#comment-17201920 ] Akira Ajisaka commented on HADOOP-17255: bq. It wouldn't work if the keystore is in a HDFS encryption zone (it would end up in a recursive loop). Storing key store in unencrypted HDFS could in theory work, but transmitting unencrypted key store compromises security. Yes. Thank you for your explanation. Note: * I understand there are very few (or no) people using Hadoop KMS and the keystore is in unencrypted HDFS. * Therefore it may not work even if this PR is merged. * I'll use Ranger KMS instead because CDP/EMR recommend using it. * Probably there are many users, so the quality of Ranger KMS is better than that of Hadoop KMS. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195743#comment-17195743 ] Wei-Chiu Chuang commented on HADOOP-17255: -- That said, the PR itself looks good to me. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195688#comment-17195688 ] Wei-Chiu Chuang commented on HADOOP-17255: -- So.. my impression came from this jira HADOOP-15412 where at the time, KMS wouldn't even start. Maybe things have changed so that it now starts fine but only fails after. Aside from that, there's security implication. Quoting myself in the jira: br. Even if you use a shared file system (like NFS?) you still need to make sure the network communication is authentication and encrypted. It wouldn't work if the keystore is in a HDFS encryption zone (it would end up in a recursive loop). Storing key store in unencrypted HDFS could in theory work, but transmitting unencrypted key store compromises security. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17195181#comment-17195181 ] Akira Ajisaka commented on HADOOP-17255: Hi [~hexiaoqiao], I think this is not a blocker. Updated the target version from 3.2.2 to 3.2.3. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17194775#comment-17194775 ] Xiaoqiao He commented on HADOOP-17255: -- [~aajisaka][~weichiu] Thanks for your works. Do we have the final conclusion to resolved and if is it release-blocker to release 3.2.2 since it tags target version include 3.2.2. > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[
https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193369#comment-17193369
]
Akira Ajisaka commented on HADOOP-17255:
Thank you [~weichiu] for your comment.
The credential provider document
([https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html#Provider_Types])
says how to configure keystore provider for HDFS.
{quote}To wrap filesystem URIs with a jceks URI follow these steps:
1. Take a filesystem URI such as hdfs://namenode:9001/users/alice/secrets.jceks
2. Place jceks:// in front of the URL:
jceks://hdfs://namenode:9001/users/alice/secrets.jceks
3. Replace the second :// string with an @ symbol:
jceks://hdfs@namenode:9001/users/alice/secrets.jceks
{quote}
Therefore I thought JavaKeyStoreProvider is supposed to work if the keystore is
in HDFS.
If it won't work on HDFS, can we add warn or error message if the keystore
provider is HDFS?
> JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
> --
>
> Key: HADOOP-17255
> URL: https://issues.apache.org/jira/browse/HADOOP-17255
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws
> FileNotFoundException if the src does not exist. However,
> JavaKeyStoreProvider#renameOrFail calls the old rename API. In
> DistributedFileSystem, the old API returns false if the src does not exist.
> That way JavaKeyStoreProvider fails to create a new key if the keystore is
> HDFS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193227#comment-17193227 ] Wei-Chiu Chuang commented on HADOOP-17255: -- I was always under the impression that key store isn't suppose to be in HDFS because it won't work. Even if you change the rename() call, there are other FileSystem calls invoked by JavaKeyStore that wouldn't work. Am I right? > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-17255) JavaKeyStoreProvider fails to create a new key if the keystore is HDFS
[ https://issues.apache.org/jira/browse/HADOOP-17255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192809#comment-17192809 ] Akira Ajisaka commented on HADOOP-17255: Stacktrace: https://gist.github.com/aajisaka/bc0dae8447b94af2e978e87cff3e7c49 > JavaKeyStoreProvider fails to create a new key if the keystore is HDFS > -- > > Key: HADOOP-17255 > URL: https://issues.apache.org/jira/browse/HADOOP-17255 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Akira Ajisaka >Priority: Critical > > The caller of JavaKeyStoreProvider#renameOrFail assumes that it throws > FileNotFoundException if the src does not exist. However, > JavaKeyStoreProvider#renameOrFail calls the old rename API. In > DistributedFileSystem, the old API returns false if the src does not exist. > That way JavaKeyStoreProvider fails to create a new key if the keystore is > HDFS. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
