[jira] [Commented] (DRILL-8353) Format plugin for Delta Lake
[ https://issues.apache.org/jira/browse/DRILL-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642321#comment-17642321 ] ASF GitHub Bot commented on DRILL-8353: --- vvysotskyi commented on PR #2702: URL: https://github.com/apache/drill/pull/2702#issuecomment-1334839612 Hi @kmatt, no, it is not supported yet, but will be added in the near future. The version will be specified using the table function. Here is the example query for it: ```sql SELECT * FROM table(dfs.delta.`/tmp/delta-table`(type => 'delta', version => 0)); ``` > Format plugin for Delta Lake > > > Key: DRILL-8353 > URL: https://issues.apache.org/jira/browse/DRILL-8353 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.2 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: Future > > > Implement format plugin for Delta Lake. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8353) Format plugin for Delta Lake
[ https://issues.apache.org/jira/browse/DRILL-8353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642248#comment-17642248 ] ASF GitHub Bot commented on DRILL-8353: --- kmatt commented on PR #2702: URL: https://github.com/apache/drill/pull/2702#issuecomment-1334708491 @vvysotskyi Does this support VERSION AS OF queries? https://docs.delta.io/latest/quick-start.html#read-older-versions-of-data-using-time-travel Ex: `SELECT * FROM dfs.delta.`/tmp/delta-table` VERSION AS OF 0;` > Format plugin for Delta Lake > > > Key: DRILL-8353 > URL: https://issues.apache.org/jira/browse/DRILL-8353 > Project: Apache Drill > Issue Type: New Feature >Affects Versions: 1.20.2 >Reporter: Vova Vysotskyi >Assignee: Vova Vysotskyi >Priority: Major > Fix For: Future > > > Implement format plugin for Delta Lake. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DRILL-8359) Add mount and unmount command support to the filesystem plugin
[
https://issues.apache.org/jira/browse/DRILL-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642177#comment-17642177
]
ASF GitHub Bot commented on DRILL-8359:
---
cgivre commented on code in PR #2713:
URL: https://github.com/apache/drill/pull/2713#discussion_r1037418363
##
contrib/storage-splunk/pom.xml:
##
@@ -42,7 +42,7 @@
com.splunk
splunk
- 1.9.1
+ 1.9.2
Review Comment:
Do we want to include the Splunk update on this PR?
##
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemConfig.java:
##
@@ -53,18 +53,23 @@ public class FileSystemConfig extends StoragePluginConfig {
public static final String NAME = "file";
private final String connection;
+ private final String[] mountCommand, unmountCommand;
Review Comment:
Is there a reason we're using `String[]` and not an `ArrayList` here?
##
exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/FileSystemPlugin.java:
##
@@ -282,4 +286,82 @@ public Set
getOptimizerRules(OptimizerRulesContext optimiz
public Configuration getFsConf() {
return new Configuration(fsConf);
}
+
+ /**
+ * Runs the configured mount command if the mounted flag is unset
+ * @return true if the configured mount command was executed
+ */
+ private synchronized boolean mount() {
+String[] mountCmd = config.getMountCommand();
+if (ArrayUtils.isEmpty(mountCmd)) {
+ return false;
+}
+try {
+ Process proc = Runtime.getRuntime().exec(mountCmd);
+ if (proc.waitFor() != 0) {
+String stderrOutput = IOUtils.toString(proc.getErrorStream(),
StandardCharsets.UTF_8);
+throw new IOException(stderrOutput);
+ }
+ logger.info("The mount command for plugin {} succeeded.", getName());
+ return true;
+} catch (IOException | InterruptedException e) {
+ logger.error("The mount command for plugin {} failed.", getName(), e);
+ throw UserException.pluginError(e)
+.message("The mount command for plugin %s failed.", getName())
+.build(logger);
+}
+ }
+
+ /**
+ * Runs the configured unmount command if the mounted flag is set
+ * @return true if the configured unmount command was executed
+ */
+ private synchronized boolean unmount() {
+String[] unmountCmd = config.getUnmountCommand();
+if (ArrayUtils.isEmpty(unmountCmd)) {
Review Comment:
See above comment about `String[]` vs `ArrayList`.
> Add mount and unmount command support to the filesystem plugin
> --
>
> Key: DRILL-8359
> URL: https://issues.apache.org/jira/browse/DRILL-8359
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - File
>Affects Versions: 1.20.2
>Reporter: James Turton
>Assignee: James Turton
>Priority: Minor
> Fix For: 2.0.0
>
>
> This Jira proposes optional mount and unmount commands in the filesystem
> plugin with the goal of enabling the dynamic definition of filesystem mounts
> in the storage configuration. It is mainly anticpiated that network and cloud
> filesystems that have FUSE drivers (sshfs, davfs, rclone, ...) will be used
> in this way but local device mounts and image/loop device mounts (ISO, IMG,
> squashfs, etc.) might also be of interest. Filesystems that can be mounted in
> this way become queryable by Drill cluster without burden of dedicated
> storage plugin development.
> The provided commands are executed in their own processes by the host OS and
> run under the OS user that is running the Drill JVM. The mount command will
> be executed when an enabled plugin is initialised (something that is done
> lazily) and whenever it transitions from disabled to enabled. The provided
> unmount command will be executed whenever a plugin transitions from enabled
> to disabled and when the Drillbit shuts down while the plugin has been
> initialised and is enabled.
> Example using udisks on Linux to mount and unmount an image of an ext4
> filesystem.
> {code:java}
> {
> "type" : "file",
> "connection" : "file:///",
> "mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
> udisksctl mount -b /dev/loop0" ],
> "unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
> udisksctl loop-delete -b /dev/loop0" ],
> "workspaces" : {
> ...{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8359) Add mount and unmount command support to the filesystem plugin
[
https://issues.apache.org/jira/browse/DRILL-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17642074#comment-17642074
]
ASF GitHub Bot commented on DRILL-8359:
---
jnturton opened a new pull request, #2713:
URL: https://github.com/apache/drill/pull/2713
# [DRILL-8359](https://issues.apache.org/jira/browse/DRILL-8359): Add mount
and unmount command support to the filesystem plugin
## Description
Optional mount and unmount commands are added toi the filesystem plugin with
the goal of enabling the dynamic definition of filesystem mounts in the storage
configuration. It is mainly anticpiated that network and cloud filesystems that
have FUSE drivers (sshfs, davfs, rclone, ...) will be used in this way but
local device mounts and image/loop device mounts (ISO, IMG, squashfs, etc.)
might also be of interest. Filesystems that can be mounted in this way become
queryable by Drill without burden of dedicated storage plugin development.
The provided commands are executed in their own processes by the host OS and
run under the OS user that is running the Drill JVM. The mount command will be
executed when an enabled plugin is initialised (something that is done lazily)
and when it transitions from disabled to enabled. The provided unmount command
will be executed whenever a plugin transitions from enabled to disabled and
when the Drillbit shuts down while the plugin is enabled.
Example using udisks on Linux to mount and unmount an image of an ext4
filesystem.
```
{
"type" : "file",
"connection" : "file:///",
"mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
udisksctl mount -b /dev/loop0" ],
"unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
udisksctl loop-delete -b /dev/loop0" ],
"workspaces" : {
...
```
## Documentation
New sections to be added to the filesystem doc page.
## Testing
New unit test TestMountCommand.
Manual test of mount commands through different sequences of startup,
enable, disable, shutdown.
> Add mount and unmount command support to the filesystem plugin
> --
>
> Key: DRILL-8359
> URL: https://issues.apache.org/jira/browse/DRILL-8359
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - File
>Affects Versions: 1.20.2
>Reporter: James Turton
>Assignee: James Turton
>Priority: Minor
> Fix For: 2.0.0
>
>
> This Jira proposes optional mount and unmount commands in the filesystem
> plugin with the goal of enabling the dynamic definition of filesystem mounts
> in the storage configuration. It is mainly anticpiated that network and cloud
> filesystems that have FUSE drivers (sshfs, davfs, rclone, ...) will be used
> in this way but local device mounts and image/loop device mounts (ISO, IMG,
> squashfs, etc.) might also be of interest. Filesystems that can be mounted in
> this way become queryable by Drill cluster without burden of dedicated
> storage plugin development.
> The provided commands are executed in their own processes by the host OS and
> run under the OS user that is running the Drill JVM. The mount command will
> be executed when an enabled plugin is initialised (something that is done
> lazily) and whenever it transitions from disabled to enabled. The provided
> unmount command will be executed whenever a plugin transitions from enabled
> to disabled and when the Drillbit shuts down while the plugin has been
> initialised and is enabled.
> Example using udisks on Linux to mount and unmount an image of an ext4
> filesystem.
> {code:java}
> {
> "type" : "file",
> "connection" : "file:///",
> "mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
> udisksctl mount -b /dev/loop0" ],
> "unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
> udisksctl loop-delete -b /dev/loop0" ],
> "workspaces" : {
> ...{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8359) Add mount and unmount command support to the filesystem plugin
[
https://issues.apache.org/jira/browse/DRILL-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8359:
Description:
This Jira proposes optional mount and unmount commands in the filesystem plugin
with the goal of enabling the dynamic definition of filesystem mounts in the
storage configuration. It is mainly anticpiated that network and cloud
filesystems that have FUSE drivers (sshfs, davfs, rclone, ...) will be used in
this way but local device mounts and image/loop device mounts (ISO, IMG,
squashfs, etc.) might also be of interest. Filesystems that can be mounted in
this way become queryable by Drill cluster without burden of dedicated storage
plugin development.
The provided commands are executed in their own processes by the host OS and
run under the OS user that is running the Drill JVM. The mount command will be
executed when an enabled plugin is initialised (something that is done lazily)
and whenever it transitions from disabled to enabled. The provided unmount
command will be executed whenever a plugin transitions from enabled to disabled
and when the Drillbit shuts down while the plugin has been initialised and is
enabled.
Example using udisks on Linux to mount and unmount an image of an ext4
filesystem.
{code:java}
{
"type" : "file",
"connection" : "file:///",
"mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
udisksctl mount -b /dev/loop0" ],
"unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
udisksctl loop-delete -b /dev/loop0" ],
"workspaces" : {
...{code}
was:
This Jira proposes optional mount and unmount commands in the filesystem plugin
with the goal of enabling the dynamic definition of filesystem mounts in the
storage configuration. It is mainly anticpiated that network and cloud
filesystems that have FUSE drivers will be used in this way but local device
mounts and image/loop device mounts (ISO, IMG, squashfs, etc.) might also be of
interest.
The provided commands are executed in their own processes by the host OS and
run under the OS user that is running the Drill JVM. The mount command will be
executed when an enabled plugin is initialised (something that is done lazily)
and whenever it transitions from disabled to enabled. The provided unmount
command will be executed whenever a plugin transitions from enabled to disabled
and when the Drillbit shuts down while the plugin has been initialised and is
enabled.
Example that
{code:java}
{
"type" : "file",
"connection" : "file:///",
"mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
udisksctl mount -b /dev/loop0" ],
"unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
udisksctl loop-delete -b /dev/loop0" ],
"workspaces" : { {code}
> Add mount and unmount command support to the filesystem plugin
> --
>
> Key: DRILL-8359
> URL: https://issues.apache.org/jira/browse/DRILL-8359
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - File
>Affects Versions: 1.20.2
>Reporter: James Turton
>Assignee: James Turton
>Priority: Minor
> Fix For: 2.0.0
>
>
> This Jira proposes optional mount and unmount commands in the filesystem
> plugin with the goal of enabling the dynamic definition of filesystem mounts
> in the storage configuration. It is mainly anticpiated that network and cloud
> filesystems that have FUSE drivers (sshfs, davfs, rclone, ...) will be used
> in this way but local device mounts and image/loop device mounts (ISO, IMG,
> squashfs, etc.) might also be of interest. Filesystems that can be mounted in
> this way become queryable by Drill cluster without burden of dedicated
> storage plugin development.
> The provided commands are executed in their own processes by the host OS and
> run under the OS user that is running the Drill JVM. The mount command will
> be executed when an enabled plugin is initialised (something that is done
> lazily) and whenever it transitions from disabled to enabled. The provided
> unmount command will be executed whenever a plugin transitions from enabled
> to disabled and when the Drillbit shuts down while the plugin has been
> initialised and is enabled.
> Example using udisks on Linux to mount and unmount an image of an ext4
> filesystem.
> {code:java}
> {
> "type" : "file",
> "connection" : "file:///",
> "mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
> udisksctl mount -b /dev/loop0" ],
> "unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
> udisksctl loop-delete -b /dev/loop0" ],
> "workspaces" : {
> ...{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DRILL-8359) Add mount and unmount command support to the filesystem plugin
[
https://issues.apache.org/jira/browse/DRILL-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Turton updated DRILL-8359:
Description:
This Jira proposes optional mount and unmount commands in the filesystem plugin
with the goal of enabling the dynamic definition of filesystem mounts in the
storage configuration. It is mainly anticpiated that network and cloud
filesystems that have FUSE drivers will be used in this way but local device
mounts and image/loop device mounts (ISO, IMG, squashfs, etc.) might also be of
interest.
The provided commands are executed in their own processes by the host OS and
run under the OS user that is running the Drill JVM. The mount command will be
executed when an enabled plugin is initialised (something that is done lazily)
and whenever it transitions from disabled to enabled. The provided unmount
command will be executed whenever a plugin transitions from enabled to disabled
and when the Drillbit shuts down while the plugin has been initialised and is
enabled.
Example that
{code:java}
{
"type" : "file",
"connection" : "file:///",
"mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
udisksctl mount -b /dev/loop0" ],
"unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
udisksctl loop-delete -b /dev/loop0" ],
"workspaces" : { {code}
was:
This Jira proposes optional mount and unmount commands in the filesystem plugin
with the goal of enabling the dynamic definition of filesystem mounts in the
storage configuration. It is mainly anticpiated that network and cloud
filesystems that have FUSE drivers will be used in this way but local device
mounts and image/loop device mounts (ISO, IMG, squashfs, etc.) might also be of
interest.
The provided commands are executed in their own processes by the host OS and
run under the OS user that is running the Drill JVM. The mount command will be
executed when an enabled plugin is initialised (something that is done lazily)
and whenever it transitions from disabled to enabled. The provided unmount
command will be executed whenever a plugin transitions from enabled to disabled
and when the Drillbit shuts down while the plugin has been initialised and is
enabled.
Example. TODO
{code:java}
{
"type": "file",
"connection": "file:///",
"mountCommand": "touch /tmp/foo.drill",
"unmountCommand": "rm /tmp/foo.drill",
"workspaces": { ...
{code}
> Add mount and unmount command support to the filesystem plugin
> --
>
> Key: DRILL-8359
> URL: https://issues.apache.org/jira/browse/DRILL-8359
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - File
>Affects Versions: 1.20.2
>Reporter: James Turton
>Assignee: James Turton
>Priority: Minor
> Fix For: 2.0.0
>
>
> This Jira proposes optional mount and unmount commands in the filesystem
> plugin with the goal of enabling the dynamic definition of filesystem mounts
> in the storage configuration. It is mainly anticpiated that network and cloud
> filesystems that have FUSE drivers will be used in this way but local device
> mounts and image/loop device mounts (ISO, IMG, squashfs, etc.) might also be
> of interest.
>
> The provided commands are executed in their own processes by the host OS and
> run under the OS user that is running the Drill JVM. The mount command will
> be executed when an enabled plugin is initialised (something that is done
> lazily) and whenever it transitions from disabled to enabled. The provided
> unmount command will be executed whenever a plugin transitions from enabled
> to disabled and when the Drillbit shuts down while the plugin has been
> initialised and is enabled.
>
> Example that
> {code:java}
> {
> "type" : "file",
> "connection" : "file:///",
> "mountCommand" : [ "sh", "-c", "udisksctl loop-setup -f /tmp/test.img &&
> udisksctl mount -b /dev/loop0" ],
> "unmountCommand" : [ "sh", "-c", "udisksctl unmount -b /dev/loop0 &&
> udisksctl loop-delete -b /dev/loop0" ],
> "workspaces" : { {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DRILL-8358) Storage plugin for querying other Apache Drill clusters
[
https://issues.apache.org/jira/browse/DRILL-8358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641892#comment-17641892
]
ASF GitHub Bot commented on DRILL-8358:
---
jnturton commented on code in PR #2709:
URL: https://github.com/apache/drill/pull/2709#discussion_r1037061461
##
contrib/storage-drill/src/main/java/org/apache/drill/exec/store/drill/plugin/DrillStoragePluginConfig.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.drill.plugin;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.calcite.avatica.ConnectStringParser;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.config.DrillProperties;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.logical.security.CredentialsProvider;
+import org.apache.drill.common.logical.security.PlainCredentialsProvider;
+import org.apache.drill.exec.client.DrillClient;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.rpc.RpcException;
+
+import java.sql.SQLException;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Properties;
+
+@JsonTypeName(DrillStoragePluginConfig.NAME)
+public class DrillStoragePluginConfig extends StoragePluginConfig {
+ public static final String NAME = "drill";
+ public static final String CONNECTION_STRING_PREFIX = "jdbc:drill:";
+
+ private static final String DEFAULT_QUOTING_IDENTIFIER = "`";
+
+ private final String connection;
+ private final Properties properties;
+
+ @JsonCreator
+ public DrillStoragePluginConfig(
+ @JsonProperty("connection") String connection,
+ @JsonProperty("properties") Properties properties,
+ @JsonProperty("credentialsProvider") CredentialsProvider
credentialsProvider) {
+super(getCredentialsProvider(credentialsProvider), credentialsProvider ==
null);
+this.connection = connection;
+this.properties = Optional.ofNullable(properties).orElse(new Properties());
+ }
+
+ private DrillStoragePluginConfig(DrillStoragePluginConfig that,
+CredentialsProvider credentialsProvider) {
+super(getCredentialsProvider(credentialsProvider),
+ credentialsProvider == null, that.authMode);
+this.connection = that.connection;
+this.properties = that.properties;
+ }
+
+ @JsonProperty("connection")
+ public String getConnection() {
+return connection;
+ }
+
+ @JsonProperty("properties")
+ public Properties getProperties() {
+return properties;
+ }
+
+ private static CredentialsProvider
getCredentialsProvider(CredentialsProvider credentialsProvider) {
+return credentialsProvider != null ? credentialsProvider :
PlainCredentialsProvider.EMPTY_CREDENTIALS_PROVIDER;
+ }
+
+ @JsonIgnore
+ public String getIdentifierQuoteString() {
+return properties.getProperty(DrillProperties.QUOTING_IDENTIFIERS,
DEFAULT_QUOTING_IDENTIFIER);
+ }
+
+ @Override
+ public DrillStoragePluginConfig updateCredentialProvider(CredentialsProvider
credentialsProvider) {
+return new DrillStoragePluginConfig(this, credentialsProvider);
+ }
+
+ @JsonIgnore
+ public DrillClient getDrillClient(String userName, BufferAllocator
allocator) {
+try {
+ String urlSuffix =
connection.substring(CONNECTION_STRING_PREFIX.length());
+ Properties props = ConnectStringParser.parse(urlSuffix, properties);
+ props.putAll(credentialsProvider.getUserCredentials(userName));
Review Comment:
This getUserCredentials(String username) method is meant to fetch
per-query-user credentials for plugins that are in user translation auth mode
while the nullary method getUserCredentials() is meant for shared credentials.
Only the plain and Vault providers currently support per-user credentials. You
can see some logic for deciding which to call (via UsernamePasswordCredentials
objects) in JdbcStorageConfig on line 142.
Those APIs wound up being a
[jira] [Commented] (DRILL-8358) Storage plugin for querying other Apache Drill clusters
[
https://issues.apache.org/jira/browse/DRILL-8358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641891#comment-17641891
]
ASF GitHub Bot commented on DRILL-8358:
---
jnturton commented on code in PR #2709:
URL: https://github.com/apache/drill/pull/2709#discussion_r1037061461
##
contrib/storage-drill/src/main/java/org/apache/drill/exec/store/drill/plugin/DrillStoragePluginConfig.java:
##
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.store.drill.plugin;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonTypeName;
+import org.apache.calcite.avatica.ConnectStringParser;
+import org.apache.drill.common.config.DrillConfig;
+import org.apache.drill.common.config.DrillProperties;
+import org.apache.drill.common.logical.StoragePluginConfig;
+import org.apache.drill.common.logical.security.CredentialsProvider;
+import org.apache.drill.common.logical.security.PlainCredentialsProvider;
+import org.apache.drill.exec.client.DrillClient;
+import org.apache.drill.exec.memory.BufferAllocator;
+import org.apache.drill.exec.rpc.RpcException;
+
+import java.sql.SQLException;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Properties;
+
+@JsonTypeName(DrillStoragePluginConfig.NAME)
+public class DrillStoragePluginConfig extends StoragePluginConfig {
+ public static final String NAME = "drill";
+ public static final String CONNECTION_STRING_PREFIX = "jdbc:drill:";
+
+ private static final String DEFAULT_QUOTING_IDENTIFIER = "`";
+
+ private final String connection;
+ private final Properties properties;
+
+ @JsonCreator
+ public DrillStoragePluginConfig(
+ @JsonProperty("connection") String connection,
+ @JsonProperty("properties") Properties properties,
+ @JsonProperty("credentialsProvider") CredentialsProvider
credentialsProvider) {
+super(getCredentialsProvider(credentialsProvider), credentialsProvider ==
null);
+this.connection = connection;
+this.properties = Optional.ofNullable(properties).orElse(new Properties());
+ }
+
+ private DrillStoragePluginConfig(DrillStoragePluginConfig that,
+CredentialsProvider credentialsProvider) {
+super(getCredentialsProvider(credentialsProvider),
+ credentialsProvider == null, that.authMode);
+this.connection = that.connection;
+this.properties = that.properties;
+ }
+
+ @JsonProperty("connection")
+ public String getConnection() {
+return connection;
+ }
+
+ @JsonProperty("properties")
+ public Properties getProperties() {
+return properties;
+ }
+
+ private static CredentialsProvider
getCredentialsProvider(CredentialsProvider credentialsProvider) {
+return credentialsProvider != null ? credentialsProvider :
PlainCredentialsProvider.EMPTY_CREDENTIALS_PROVIDER;
+ }
+
+ @JsonIgnore
+ public String getIdentifierQuoteString() {
+return properties.getProperty(DrillProperties.QUOTING_IDENTIFIERS,
DEFAULT_QUOTING_IDENTIFIER);
+ }
+
+ @Override
+ public DrillStoragePluginConfig updateCredentialProvider(CredentialsProvider
credentialsProvider) {
+return new DrillStoragePluginConfig(this, credentialsProvider);
+ }
+
+ @JsonIgnore
+ public DrillClient getDrillClient(String userName, BufferAllocator
allocator) {
+try {
+ String urlSuffix =
connection.substring(CONNECTION_STRING_PREFIX.length());
+ Properties props = ConnectStringParser.parse(urlSuffix, properties);
+ props.putAll(credentialsProvider.getUserCredentials(userName));
Review Comment:
This getUserCredentials(String username) method is meant to fetch
per-query-user credentials for plugins that are in user translation auth mode
while the nullary method getUserCredentials() is meant for shared credentials.
Only the plan and Vault providers currently support per-user credentials. You
can see some logic for deciding which to call (via UsernamePasswordCredentials
objects) in JdbcStorageConfig on line 142.
Those APIs wound up being a
