[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727209#comment-16727209 ] Bridget Bevens commented on DRILL-6662: --- Hi [~KazydubB], I've added a section to the s3 storage plugin config doc here: [https://drill.apache.org/docs/s3-storage-plugin/#using-an-external-provider-for-credentials] Please have a look and let me know if I need to make any changes. Thanks! Bridget > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724177#comment-16724177 ] Denys Ordynskiy commented on DRILL-6662: Successfully tested on Drill storage plugin and Drill core-site.xml > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting, ready-to-commit > Fix For: 1.15.0 > > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584765#comment-16584765
]
ASF GitHub Bot commented on DRILL-6662:
---
asfgit closed pull request #1419: DRILL-6662: Access AWS access key ID and
secret access key using Cred…
URL: https://github.com/apache/drill/pull/1419
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/distribution/src/resources/core-site-example.xml
b/distribution/src/resources/core-site-example.xml
index 854e54dbe67..c7225a14784 100644
--- a/distribution/src/resources/core-site-example.xml
+++ b/distribution/src/resources/core-site-example.xml
@@ -30,4 +30,18 @@
ENTER_YOUR_SECRETKEY
+
+
+
+
+
+
diff --git
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
index b1f41a41451..cb66913ad13 100644
---
a/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
+++
b/exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
@@ -20,6 +20,7 @@
import static
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.DEFAULT_WS_NAME;
import java.io.IOException;
+import java.net.URI;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
@@ -77,6 +78,10 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
fsConf.set("fs.classpath.impl", ClassPathFileSystem.class.getName());
fsConf.set("fs.drill-local.impl",
LocalSyncableFileSystem.class.getName());
+ if (isS3Connection(fsConf)) {
+handleS3Credentials(fsConf);
+ }
+
formatCreator = newFormatCreator(config, context, fsConf);
List matchers = new ArrayList<>();
formatPluginsByConfig = new HashMap<>();
@@ -104,6 +109,33 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
}
}
+ private boolean isS3Connection(Configuration conf) {
+URI uri = FileSystem.getDefaultUri(conf);
+return uri.getScheme().equals("s3a");
+ }
+
+ /**
+ * Retrieve secret and access keys from configured (with
+ * {@link
org.apache.hadoop.security.alias.CredentialProviderFactory#CREDENTIAL_PROVIDER_PATH}
property)
+ * credential providers and set it into {@code conf}. If provider path is
not configured or credential
+ * is absent in providers, it will conditionally fallback to configuration
setting. The fallback will occur unless
+ * {@link
org.apache.hadoop.security.alias.CredentialProvider#CLEAR_TEXT_FALLBACK} is set
to {@code false}.
+ *
+ * @param conf {@code Configuration} which will be updated with credentials
from provider
+ * @throws IOException thrown if a credential cannot be retrieved from
provider
+ */
+ private void handleS3Credentials(Configuration conf) throws IOException {
+String[] credentialKeys = {"fs.s3a.secret.key", "fs.s3a.access.key"};
+for (String key : credentialKeys) {
+ char[] credentialChars = conf.getPassword(key);
+ if (credentialChars == null) {
+logger.warn(String.format("Property '%s' is absent.", key));
+ } else {
+conf.set(key, String.valueOf(credentialChars));
+ }
+}
+ }
+
/**
* Creates a new FormatCreator instance.
*
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Access AWS access key ID and secret access key using Credential Provider API
> for S3 storage plugin
> --
>
> Key: DRILL-6662
> URL: https://issues.apache.org/jira/browse/DRILL-6662
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting, ready-to-commit
> Fix For: 1.15.0
>
>
> Hadoop provides [CredentialProvider
> API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]]
> which allows passwords and other sensitive secrets to be stored in an
> external provider rather than in configuration files in plaintext.
> Currently S3 storage plugin is accessing passwords, namely
> 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in
> Configuration with
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582372#comment-16582372 ] ASF GitHub Bot commented on DRILL-6662: --- KazydubB commented on issue #1419: DRILL-6662: Access AWS access key ID and secret access key using Cred… URL: https://github.com/apache/drill/pull/1419#issuecomment-413512145 @arina-ielchiieva I have addressed review comments. Could you take a look, please? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582373#comment-16582373 ] ASF GitHub Bot commented on DRILL-6662: --- arina-ielchiieva commented on issue #1419: DRILL-6662: Access AWS access key ID and secret access key using Cred… URL: https://github.com/apache/drill/pull/1419#issuecomment-413512174 +1, LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582371#comment-16582371
]
ASF GitHub Bot commented on DRILL-6662:
---
arina-ielchiieva commented on a change in pull request #1419: DRILL-6662:
Access AWS access key ID and secret access key using Cred…
URL: https://github.com/apache/drill/pull/1419#discussion_r210559817
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
##
@@ -104,6 +109,33 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
}
}
+ private boolean isS3Connection(Configuration conf) {
+URI uri = FileSystem.getDefaultUri(conf);
+return uri.getScheme().equals("s3a");
+ }
+
+ /**
+ * Retrieve secret and access keys from configured (with
+ * {@link
org.apache.hadoop.security.alias.CredentialProviderFactory#CREDENTIAL_PROVIDER_PATH}
property)
+ * credential providers and set it into {@code conf}. If provider path is
not configured or credential
+ * is absent in providers, it will conditionally fallback to configuration
setting. The fallback will occur unless
+ * {@link
org.apache.hadoop.security.alias.CredentialProvider#CLEAR_TEXT_FALLBACK} is set
to {@code false}.
+ *
+ * @param conf {@code Configuration} which will be updated with credentials
from provider
+ * @throws IOException thrown if a credential cannot be retrieved from
provider
+ */
+ private void handleS3Credentials(Configuration conf) throws IOException {
+String[] credentialKeys = {"fs.s3a.secret.key", "fs.s3a.access.key"};
Review comment:
In this case, please leave as is.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Access AWS access key ID and secret access key using Credential Provider API
> for S3 storage plugin
> --
>
> Key: DRILL-6662
> URL: https://issues.apache.org/jira/browse/DRILL-6662
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Hadoop provides [CredentialProvider
> API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]]
> which allows passwords and other sensitive secrets to be stored in an
> external provider rather than in configuration files in plaintext.
> Currently S3 storage plugin is accessing passwords, namely
> 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in
> Configuration with get() method. To give users an ability to remove clear
> text passwords for S3 from configuration files Configuration.getPassword()
> method should be used, given they configure
> 'hadoop.security.credential.provider.path' property which points to a file
> containing encrypted passwords instead of configuring two aforementioned
> properties.
> By using this approach, credential providers will be checked first and if the
> secret is not provided or providers are not configured there will be a
> fallback to secrets configured in clear text (unless
> 'hadoop.security.credential.clear-text-fallback' is configured to be
> "false"), thus making new change backwards-compatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582369#comment-16582369
]
ASF GitHub Bot commented on DRILL-6662:
---
KazydubB commented on a change in pull request #1419: DRILL-6662: Access AWS
access key ID and secret access key using Cred…
URL: https://github.com/apache/drill/pull/1419#discussion_r210559211
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
##
@@ -104,6 +109,33 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
}
}
+ private boolean isS3Connection(Configuration conf) {
+URI uri = FileSystem.getDefaultUri(conf);
+return uri.getScheme().equals("s3a");
+ }
+
+ /**
+ * Retrieve secret and access keys from configured (with
+ * {@link
org.apache.hadoop.security.alias.CredentialProviderFactory#CREDENTIAL_PROVIDER_PATH}
property)
+ * credential providers and set it into {@code conf}. If provider path is
not configured or credential
+ * is absent in providers, it will conditionally fallback to configuration
setting. The fallback will occur unless
+ * {@link
org.apache.hadoop.security.alias.CredentialProvider#CLEAR_TEXT_FALLBACK} is set
to {@code false}.
+ *
+ * @param conf {@code Configuration} which will be updated with credentials
from provider
+ * @throws IOException thrown if a credential cannot be retrieved from
provider
+ */
+ private void handleS3Credentials(Configuration conf) throws IOException {
+String[] credentialKeys = {"fs.s3a.secret.key", "fs.s3a.access.key"};
Review comment:
I am aware of the Constants, but artifact (hadoop-aws), containing this
class is not among the module's dependencies (however it is present in
distribution's (compile-scope) and drill-root's (test-scope) dependencies). Is
there a need to add the dependency?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Access AWS access key ID and secret access key using Credential Provider API
> for S3 storage plugin
> --
>
> Key: DRILL-6662
> URL: https://issues.apache.org/jira/browse/DRILL-6662
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Hadoop provides [CredentialProvider
> API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]]
> which allows passwords and other sensitive secrets to be stored in an
> external provider rather than in configuration files in plaintext.
> Currently S3 storage plugin is accessing passwords, namely
> 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in
> Configuration with get() method. To give users an ability to remove clear
> text passwords for S3 from configuration files Configuration.getPassword()
> method should be used, given they configure
> 'hadoop.security.credential.provider.path' property which points to a file
> containing encrypted passwords instead of configuring two aforementioned
> properties.
> By using this approach, credential providers will be checked first and if the
> secret is not provided or providers are not configured there will be a
> fallback to secrets configured in clear text (unless
> 'hadoop.security.credential.clear-text-fallback' is configured to be
> "false"), thus making new change backwards-compatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582314#comment-16582314 ] ASF GitHub Bot commented on DRILL-6662: --- arina-ielchiieva commented on a change in pull request #1419: DRILL-6662: Access AWS access key ID and secret access key using Cred… URL: https://github.com/apache/drill/pull/1419#discussion_r210540931 ## File path: distribution/src/resources/core-site-example.xml ## @@ -30,4 +30,14 @@ ENTER_YOUR_SECRETKEY + Review comment: Please comment out this section and add comment explaining that user should use one of those. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > Labels: doc-impacting > Fix For: 1.15.0 > > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16582313#comment-16582313
]
ASF GitHub Bot commented on DRILL-6662:
---
arina-ielchiieva commented on a change in pull request #1419: DRILL-6662:
Access AWS access key ID and secret access key using Cred…
URL: https://github.com/apache/drill/pull/1419#discussion_r210541907
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
##
@@ -104,6 +109,33 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
}
}
+ private boolean isS3Connection(Configuration conf) {
+URI uri = FileSystem.getDefaultUri(conf);
+return uri.getScheme().equals("s3a");
+ }
+
+ /**
+ * Retrieve secret and access keys from configured (with
+ * {@link
org.apache.hadoop.security.alias.CredentialProviderFactory#CREDENTIAL_PROVIDER_PATH}
property)
+ * credential providers and set it into {@code conf}. If provider path is
not configured or credential
+ * is absent in providers, it will conditionally fallback to configuration
setting. The fallback will occur unless
+ * {@link
org.apache.hadoop.security.alias.CredentialProvider#CLEAR_TEXT_FALLBACK} is set
to {@code false}.
+ *
+ * @param conf {@code Configuration} which will be updated with credentials
from provider
+ * @throws IOException thrown if a credential cannot be retrieved from
provider
+ */
+ private void handleS3Credentials(Configuration conf) throws IOException {
+String[] credentialKeys = {"fs.s3a.secret.key", "fs.s3a.access.key"};
Review comment:
Consider using org.apache.hadoop.fs.s3a.Contants class.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Access AWS access key ID and secret access key using Credential Provider API
> for S3 storage plugin
> --
>
> Key: DRILL-6662
> URL: https://issues.apache.org/jira/browse/DRILL-6662
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
> Labels: doc-impacting
> Fix For: 1.15.0
>
>
> Hadoop provides [CredentialProvider
> API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]]
> which allows passwords and other sensitive secrets to be stored in an
> external provider rather than in configuration files in plaintext.
> Currently S3 storage plugin is accessing passwords, namely
> 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in
> Configuration with get() method. To give users an ability to remove clear
> text passwords for S3 from configuration files Configuration.getPassword()
> method should be used, given they configure
> 'hadoop.security.credential.provider.path' property which points to a file
> containing encrypted passwords instead of configuring two aforementioned
> properties.
> By using this approach, credential providers will be checked first and if the
> secret is not provided or providers are not configured there will be a
> fallback to secrets configured in clear text (unless
> 'hadoop.security.credential.clear-text-fallback' is configured to be
> "false"), thus making new change backwards-compatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570157#comment-16570157
]
ASF GitHub Bot commented on DRILL-6662:
---
KazydubB commented on a change in pull request #1419: DRILL-6662: Access AWS
access key ID and secret access key using Cred…
URL: https://github.com/apache/drill/pull/1419#discussion_r207876730
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
##
@@ -104,6 +108,31 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
}
}
+ private boolean isS3() {
+java.net.URI uri = FileSystem.getDefaultUri(fsConf);
+return uri.getScheme().equals("s3a");
+ }
+
+ /**
+ * Retrieve secret and access keys from configured (with
+ * {@link
org.apache.hadoop.security.alias.CredentialProviderFactory#CREDENTIAL_PROVIDER_PATH}
property)
+ * credential providers and set it into {@link #fsConf}. If provider path is
not configured or credential
+ * is absent in providers, it will conditionally fallback to configuration
setting. The fallback will occur unless
+ * {@link
org.apache.hadoop.security.alias.CredentialProvider#CLEAR_TEXT_FALLBACK} is set
to false.
+ * @throws IOException thrown if a credential cannot be retrieved from
provider
+ */
+ private void handleS3Credentials() throws IOException {
+final String[] credentialKeys = {"fs.s3a.secret.key", "fs.s3a.access.key"};
+for (String key : credentialKeys) {
+ char[] credentialChars = fsConf.getPassword(key);
+ if (credentialChars != null) {
+fsConf.set(key, String.valueOf(credentialChars));
Review comment:
Hm, what do you call an "inline declaration"? In s3 storage plugin and with
hadoop credential CLI's -value (though this option is for testing because it's
unsecure) the value should be wrapped with quotes if it is designed to contain
spaces. If set in core-site.xml, the value is wrapped with tags, so any space
is considered to be intentional.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Access AWS access key ID and secret access key using Credential Provider API
> for S3 storage plugin
> --
>
> Key: DRILL-6662
> URL: https://issues.apache.org/jira/browse/DRILL-6662
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
>
> Hadoop provides [CredentialProvider
> API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]]
> which allows passwords and other sensitive secrets to be stored in an
> external provider rather than in configuration files in plaintext.
> Currently S3 storage plugin is accessing passwords, namely
> 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in
> Configuration with get() method. To give users an ability to remove clear
> text passwords for S3 from configuration files Configuration.getPassword()
> method should be used, given they configure
> 'hadoop.security.credential.provider.path' property which points to a file
> containing encrypted passwords instead of configuring two aforementioned
> properties.
> By using this approach, credential providers will be checked first and if the
> secret is not provided or providers are not configured there will be a
> fallback to secrets configured in clear text (unless
> 'hadoop.security.credential.clear-text-fallback' is configured to be
> "false"), thus making new change backwards-compatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[
https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568902#comment-16568902
]
ASF GitHub Bot commented on DRILL-6662:
---
steveloughran commented on a change in pull request #1419: DRILL-6662: Access
AWS access key ID and secret access key using Cred…
URL: https://github.com/apache/drill/pull/1419#discussion_r207686889
##
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java
##
@@ -104,6 +108,31 @@ public FileSystemPlugin(FileSystemConfig config,
DrillbitContext context, String
}
}
+ private boolean isS3() {
+java.net.URI uri = FileSystem.getDefaultUri(fsConf);
+return uri.getScheme().equals("s3a");
+ }
+
+ /**
+ * Retrieve secret and access keys from configured (with
+ * {@link
org.apache.hadoop.security.alias.CredentialProviderFactory#CREDENTIAL_PROVIDER_PATH}
property)
+ * credential providers and set it into {@link #fsConf}. If provider path is
not configured or credential
+ * is absent in providers, it will conditionally fallback to configuration
setting. The fallback will occur unless
+ * {@link
org.apache.hadoop.security.alias.CredentialProvider#CLEAR_TEXT_FALLBACK} is set
to false.
+ * @throws IOException thrown if a credential cannot be retrieved from
provider
+ */
+ private void handleS3Credentials() throws IOException {
+final String[] credentialKeys = {"fs.s3a.secret.key", "fs.s3a.access.key"};
+for (String key : credentialKeys) {
+ char[] credentialChars = fsConf.getPassword(key);
+ if (credentialChars != null) {
+fsConf.set(key, String.valueOf(credentialChars));
Review comment:
s3a connector actually calls trim() on the created string, so that you can
have some whitespace around any inline declaration. Worth retaining for
consistency
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Access AWS access key ID and secret access key using Credential Provider API
> for S3 storage plugin
> --
>
> Key: DRILL-6662
> URL: https://issues.apache.org/jira/browse/DRILL-6662
> Project: Apache Drill
> Issue Type: Improvement
>Reporter: Bohdan Kazydub
>Assignee: Bohdan Kazydub
>Priority: Major
>
> Hadoop provides [CredentialProvider
> API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]]
> which allows passwords and other sensitive secrets to be stored in an
> external provider rather than in configuration files in plaintext.
> Currently S3 storage plugin is accessing passwords, namely
> 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in
> Configuration with get() method. To give users an ability to remove clear
> text passwords for S3 from configuration files Configuration.getPassword()
> method should be used, given they configure
> 'hadoop.security.credential.provider.path' property which points to a file
> containing encrypted passwords instead of configuring two aforementioned
> properties.
> By using this approach, credential providers will be checked first and if the
> secret is not provided or providers are not configured there will be a
> fallback to secrets configured in clear text (unless
> 'hadoop.security.credential.clear-text-fallback' is configured to be
> "false"), thus making new change backwards-compatible.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568901#comment-16568901 ] Steve Loughran commented on DRILL-6662: --- bq. Drill uses older version of Hadoop (2.7.1) which does not have the utility class. no, it doesn't have the per-bucket stuff, but can be used as a copy-and-paste reference implementation of how the S3A connector itself does its password lookup. And: you are free to upgrade, at least to a later version of the 2.7.x line > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568148#comment-16568148 ] ASF GitHub Bot commented on DRILL-6662: --- KazydubB opened a new pull request #1419: DRILL-6662: Access AWS access key ID and secret access key using Cred… URL: https://github.com/apache/drill/pull/1419 …ential Provider API for S3 storage plugin This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568118#comment-16568118 ] Bohdan Kazydub commented on DRILL-6662: --- [~ste...@apache.org], thank you for suggestion, but currently Drill uses older version of Hadoop (2.7.1) which does not have the utility class. On the other hand, each plugin instance corresponds to an individual bucket, so there is a possibility to configure provider path to reference file containing bucket-specific keys. > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DRILL-6662) Access AWS access key ID and secret access key using Credential Provider API for S3 storage plugin
[ https://issues.apache.org/jira/browse/DRILL-6662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567682#comment-16567682 ] Steve Loughran commented on DRILL-6662: --- you might want to take the code from [S3A Utils|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AUtils.java#L735], which handles per-bucket secrets in the config files. That allows you to have different secrets (inc encryption keys) for different buckets > Access AWS access key ID and secret access key using Credential Provider API > for S3 storage plugin > -- > > Key: DRILL-6662 > URL: https://issues.apache.org/jira/browse/DRILL-6662 > Project: Apache Drill > Issue Type: Improvement >Reporter: Bohdan Kazydub >Assignee: Bohdan Kazydub >Priority: Major > > Hadoop provides [CredentialProvider > API|[https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html]] > which allows passwords and other sensitive secrets to be stored in an > external provider rather than in configuration files in plaintext. > Currently S3 storage plugin is accessing passwords, namely > 'fs.s3a.access.key' and 'fs.s3a.secret.key', stored in clear text in > Configuration with get() method. To give users an ability to remove clear > text passwords for S3 from configuration files Configuration.getPassword() > method should be used, given they configure > 'hadoop.security.credential.provider.path' property which points to a file > containing encrypted passwords instead of configuring two aforementioned > properties. > By using this approach, credential providers will be checked first and if the > secret is not provided or providers are not configured there will be a > fallback to secrets configured in clear text (unless > 'hadoop.security.credential.clear-text-fallback' is configured to be > "false"), thus making new change backwards-compatible. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
