This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new cfc23a1df [KYUUBI #6339] kyuubi-beeline auto constructs JDBC URL from
kyuubi-defaults.conf
cfc23a1df is described below
commit cfc23a1df92545a43f274f59c78cf3f8cdc23446
Author: hezhao2 <[email protected]>
AuthorDate: Tue Jun 25 13:19:46 2024 +0800
[KYUUBI #6339] kyuubi-beeline auto constructs JDBC URL from
kyuubi-defaults.conf
# :mag: Description
## Issue References ๐
as title
This pull request closes #6255
## Describe Your Solution ๐ง
Please include a summary of the change and which issue is fixed. Please
also include relevant motivation and context. List any dependencies that are
required for this change.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
#### Behavior Without This Pull Request :coffin:
#### Behavior With This Pull Request :tada:
#### Related Unit Tests
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6339 from zhaohehuhu/dev-0426.
Closes #6339
38b36f260 [Cheng Pan] fix
b2e3fcf3d [Cheng Pan] rewrite
Lead-authored-by: hezhao2 <[email protected]>
Co-authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
docs/client/cli/kyuubi_beeline.md | 289 +++++++++++++++++++++
docs/client/cli/kyuubi_beeline.rst | 22 --
.../main/java/org/apache/hive/beeline/BeeLine.java | 39 +--
.../KyuubiConfFileParseException.java | 29 +++
.../hs2connection/KyuubiConfFileParser.java | 205 +++++++++++++++
.../hs2connection/UserHS2ConnectionFileParser.java | 6 +-
.../hs2connection/TestKyuubiConfFileParser.java | 44 ++++
.../src/test/resources/kyuubi-defaults.conf | 40 +++
8 files changed, 633 insertions(+), 41 deletions(-)
diff --git a/docs/client/cli/kyuubi_beeline.md
b/docs/client/cli/kyuubi_beeline.md
new file mode 100644
index 000000000..3fd5208eb
--- /dev/null
+++ b/docs/client/cli/kyuubi_beeline.md
@@ -0,0 +1,289 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Kyuubi Beeline
+
+## What is Kyuubi Beeline
+
+Kyuubi Beeline is a Command Line Shell that uses JDBC driver to connect to
Kyuubi server to execute queries.
+Kyuubi Beeline is derived from
+[Hive
Beeline](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell),
+almost functionalities of Hive Beeline should be applicable to Kyuubi Beeline
as well.
+
+Note: Kyuubi Beeline removes the support of "embedded mode" because it is
coupled with Apache Hive implementation details,
+thus only "remote mode" can be used in Kyuubi Beeline.
+
+## How to use Kyuubi Beeline
+
+Run `kyuubi-beeline --help` to show help message with all options and examples.
+
+```
+Usage: kyuubi-beeline <options>.
+
+Options:
+ -u <database url> The JDBC URL to connect to.
+ -c <named url> The named JDBC URL to connect to,
+ which should be present in beeline-site.xml
+ as the value of
beeline.kyuubi.jdbc.url.<namedUrl>.
+
+ -r Reconnect to last saved connect url (in
conjunction with !save).
+ -n <username> The username to connect as.
+ -p <password> The password to connect as.
+ -d <driver class> The driver class to use.
+ -i <init file> Script file for initialization.
+ -e <query> Query that should be executed.
+ -f <exec file> Script file that should be executed.
+ -w, --password-file <file> The password file to read password from.
+ --hiveconf property=value Use value for given property.
+ --hivevar name=value Hive variable name and value.
+ This is Hive specific settings in which
variables
+ can be set at session level and referenced
in Hive
+ commands or queries.
+
+ --property-file=<property file> The file to read connection properties (url,
driver, user, password) from.
+ --color=[true|false] Control whether color is used for display.
+ --showHeader=[true|false] Show column names in query results.
+ --escapeCRLF=[true|false] Show carriage return and line feeds in query
results as escaped \r and \n.
+ --headerInterval=ROWS; The interval between which heades are
displayed.
+ --fastConnect=[true|false] Skip building table/column list for
tab-completion.
+ --autoCommit=[true|false] Enable/disable automatic transaction commit.
+ --verbose=[true|false] Show verbose error messages and debug info.
+ --showWarnings=[true|false] Display connection warnings.
+ --showDbInPrompt=[true|false] Display the current database name in the
prompt.
+ --showNestedErrs=[true|false] Display nested errors.
+ --numberFormat=[pattern] Format numbers using DecimalFormat pattern.
+ --force=[true|false] Continue running script even after errors.
+ --maxWidth=MAXWIDTH The maximum width of the terminal.
+ --maxColumnWidth=MAXCOLWIDTH The maximum width to use when displaying
columns.
+ --silent=[true|false] Be more silent.
+ --autosave=[true|false] Automatically save preferences.
+ --outputformat=<format mode> Format mode for result display.
+ The available options ars
[table|vertical|csv2|tsv2|dsv|csv|tsv|json|jsonfile].
+ Note that csv, and tsv are deprecated, use
csv2, tsv2 instead.
+
+ --incremental=[true|false] Defaults to false. When set to false, the
entire result set
+ is fetched and buffered before being
displayed, yielding optimal
+ display column sizing. When set to true,
result rows are displayed
+ immediately as they are fetched, yielding
lower latency and
+ memory usage at the price of extra display
column padding.
+ Setting --incremental=true is recommended if
you encounter an OutOfMemory
+ on the client side (due to the fetched
result set size being large).
+ Only applicable if --outputformat=table.
+
+ --incrementalBufferRows=NUMROWS The number of rows to buffer when printing
rows on stdout,
+ defaults to 1000; only applicable if
--incremental=true
+ and --outputformat=table.
+
+ --truncateTable=[true|false] Truncate table column when it exceeds length.
+ --delimiterForDSV=DELIMITER Specify the delimiter for
delimiter-separated values output format (default: |).
+ --isolation=LEVEL Set the transaction isolation level.
+ --nullemptystring=[true|false] Set to true to get historic behavior of
printing null as empty string.
+ --maxHistoryRows=MAXHISTORYROWS The maximum number of rows to store beeline
history.
+ --delimiter=DELIMITER Set the query delimiter; multi-char
delimiters are allowed, but quotation
+ marks, slashes, and -- are not allowed
(default: ;).
+
+ --convertBinaryArrayToString=[true|false]
+ Display binary column data as string or as
byte array.
+
+ --python-mode Execute python code/script.
+ -h, --help Display this message.
+
+Examples:
+ 1. Connect using simple authentication to Kyuubi Server on localhost:10009.
+ $ kyuubi-beeline -u jdbc:kyuubi://localhost:10009 -n username
+
+ 2. Connect using simple authentication to Kyuubi Server on
kyuubi.local:10009 using -n for username and -p for password.
+ $ kyuubi-beeline -n username -p password -u jdbc:kyuubi://kyuubi.local:10009
+
+ 3. Connect using Kerberos authentication with kyuubi/[email protected]
as Kyuubi Server principal(kinit is required before connection).
+ $ kyuubi-beeline -u
"jdbc:kyuubi://kyuubi.local:10009/default;kyuubiServerPrincipal=kyuubi/[email protected]"
+
+ 4. Connect using Kerberos authentication using principal and keytab directly.
+ $ kyuubi-beeline -u
"jdbc:kyuubi://kyuubi.local:10009/default;[email protected];kyuubiClientKeytab=/local/path/client.keytab;kyuubiServerPrincipal=kyuubi/[email protected]"
+
+ 5. Connect using SSL connection to Kyuubi Server on localhost:10009.
+ $ kyuubi-beeline -u
"jdbc:kyuubi://localhost:10009/default;ssl=true;sslTrustStore=/usr/local/truststore;trustStorePassword=mytruststorepassword"
+
+ 6. Connect using LDAP authentication.
+ $ kyuubi-beeline -u jdbc:kyuubi://kyuubi.local:10009/default -n
ldap-username -p ldap-password
+
+ 7. Connect using the ZooKeeper address to Kyuubi HA cluster.
+ $ kyuubi-beeline -u
"jdbc:kyuubi://zk1:2181,zk2:2181,zk3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi"
-n username
+```
+
+## Using `kyuubi-defaults.conf` to automatically connect to Kyuubi
+
+As of Kyuubi 1.10, Kyuubi Beeline adds support to use the
`kyuubi-defaults.conf` present in the `KYUUBI_CONF_DIR` to
+automatically generate a connection URL based on the configuration properties
in `kyuubi-defaults.conf` and an
+additional user configuration file. Not all the URL properties can be derived
from `kyuubi-defaults.conf` and hence
+in order to use this feature user must create a configuration file called
`beeline-hs2-connection.xml` which is a
+Hadoop XML format file. This file is used to provide user-specific connection
properties for the connection URL.
+Kyuubi Beeline looks for this configuration file in `${user.home}/.beeline/`
(UNIX-like OS) or `${user.home}\beeline\`
+directory (in case of Windows). If the file is not found in the above
locations Beeline looks for it in `HIVE_CONF_DIR`
+location and `/etc/hive/conf` in that order. Once the file is found, Kyuubi
Beeline uses `beeline-hs2-connection.xml`
+in conjunction with the `kyuubi-defaults.conf` to determine the connection URL.
+
+The URL connection properties in `beeline-hs2-connection.xml` must have the
prefix `beeline.hs2.connection.` followed by
+the URL property name. For example in order to provide the property ssl the
property key in the `beeline-hs2-connection.xml`
+should be `beeline.hs2.connection.ssl`. The sample
`beeline.hs2.connection.xml` below provides the value of user and
+password for the Beeline connection URL. In this case the rest of the
properties like Kyuubi hostname and port information,
+Kerberos configuration properties, SSL properties, transport mode, etc., are
picked up using the `kyuubi-defaults.conf`.
+If the password is empty `beeline.hs2.connection.password` property should be
removed. In most cases the below
+configuration values in `beeline-hs2-connection.xml` and the correct
`kyuubi-defaults.conf` should be sufficient to
+make the connection to the Kyuubi.
+
+```xml
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>beeline.hs2.connection.user</name>
+ <value>kyuubi</value>
+</property>
+<property>
+ <name>beeline.hs2.connection.password</name>
+ <value>kyuubi</value>
+</property>
+</configuration>
+```
+
+In case of properties which are present in both `beeline-hs2-connection.xml`
and `kyuubi-defaults.conf`, the property
+value derived from `beeline-hs2-connection.xml` takes precedence. For example
in the below `beeline-hs2-connection.xml`
+file provides the value of principal for Beeline connection in a Kerberos
enabled environment. In this case the property
+value for `beeline.hs2.connection.principal` overrides the value of
`kyuubi.kinit.principal` from `kyuubi-defaults.conf`
+as far as connection URL is concerned.
+
+```xml
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>beeline.hs2.connection.hosts</name>
+ <value>localhost:10009</value>
+</property>
+<property>
+ <name>beeline.hs2.connection.principal</name>
+ <value>kyuubi/[email protected]</value>
+</property>
+</configuration>
+```
+
+In case of properties `beeline.hs2.connection.hosts`,
`beeline.hs2.connection.hiveconf` and
+`beeline.hs2.connection.hivevar` the property value is a comma-separated list
of values. For example the following
+`beeline-hs2-connection.xml` provides the `hiveconf` and `hivevar` values in a
comma separated format.
+
+```xml
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>beeline.hs2.connection.user</name>
+ <value>kyuubi</value>
+</property>
+<property>
+ <name>beeline.hs2.connection.hiveconf</name>
+
<value>kyuubi.session.engine.initialize.timeout=PT3M,kyuubi.engine.type=SPARK_SQL</value>
+</property>
+<property>
+ <name>beeline.hs2.connection.hivevar</name>
+ <value>testVarName1=value1, testVarName2=value2</value>
+</property>
+</configuration>
+```
+
+When the `beeline-hs2-connection.xml` is present and when no other arguments
are provided, Kyuubi Beeline automatically
+connects to the URL generated using configuration files. When connection
arguments (`-u`, `-n` or `-p`) are provided,
+Kyuubi Beeline uses them and does not use `beeline-hs2-connection.xml` to
automatically connect. Removing or renaming
+the `beeline-hs2-connection.xml` disables this feature.
+
+## Using beeline-site.xml to automatically connect to HiveServer2
+
+In addition to the above method of using `kyuubi-defaults.conf` and
`beeline-hs2-connection.xml` for deriving the JDBC
+connection URL to use when connecting to Kyuubi from Kyuubi Beeline, a user
can optionally add `beeline-site.xml` to
+their classpath, and within `beeline-site.xml`, she can specify complete JDBC
URLs. A user can also specify multiple
+named URLs and use `kyuubi-beeline -c <named_url>` to connect to a specific
URL. This is particularly useful when the
+same cluster has multiple Kyuubi instances running with different
configurations. One of the named URLs is treated as
+default (which is the URL that gets used when the user simply types beeline).
An example `beeline-site.xml` is shown below:
+
+```xml
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>beeline.hs2.jdbc.url.tcpUrl</name>
+
<value>jdbc:kyuubi://localhost:10009/default;user=kyuubi;password=kyuubi</value>
+</property>
+
+<property>
+ <name>beeline.hs2.jdbc.url.httpUrl</name>
+
<value>jdbc:kyuubi://localhost:10009/default;user=kyuubi;password=kyuubi;transportMode=http;httpPath=cliservice</value>
+</property>
+
+<property>
+ <name>beeline.hs2.jdbc.url.default</name>
+ <value>tcpUrl</value>
+</property>
+</configuration>
+```
+
+In the above example, simply typing `kyuubi-beeline` opens a new JDBC
connection to
+`jdbc:kyuubi://localhost:10009/default;user=kyuubi;password=kyuubi`. If both
`beeline-site.xml` and
+`beeline-hs2-connection.xml` are present in the classpath, the final URL is
created by applying the properties specified
+in `beeline-hs2-connection.xml` on top of the URL properties derived from
beeline-site.xml. As an example consider the
+following `beeline-hs2-connection.xml`:
+
+```xml
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>beeline.hs2.connection.user</name>
+ <value>kyuubi</value>
+</property>
+<property>
+ <name>beeline.hs2.connection.password</name>
+ <value>kyuubi</value>
+</property>
+</configuration>
+```
+
+Consider the following `beeline-site.xml`:
+
+```xml
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<configuration>
+<property>
+ <name>beeline.hs2.jdbc.url.tcpUrl</name>
+ <value>jdbc:kyuubi://localhost:10009/default</value>
+</property>
+
+<property>
+ <name>beeline.hs2.jdbc.url.httpUrl</name>
+
<value>jdbc:kyuubi://localhost:10009/default;transportMode=http;httpPath=cliservice</value>
+</property>
+
+<property>
+ <name>beeline.hs2.jdbc.url.default</name>
+ <value>tcpUrl</value>
+</property>
+</configuration>
+```
+
+In the above example, simply typing `kyuubi-beeline` opens a new JDBC
connection to
+`jdbc:kyuubi://localhost:10009/default;user=kyuubi;password=kyuubi`. When the
user types `kyuubi-beeline -c httpUrl`,
+a connection is opened to
`jdbc:kyuubi://localhost:10009/default;transportMode=http;httpPath=cliservice;user=kyuubi;password=kyuubi`.
diff --git a/docs/client/cli/kyuubi_beeline.rst
b/docs/client/cli/kyuubi_beeline.rst
deleted file mode 100644
index e21781058..000000000
--- a/docs/client/cli/kyuubi_beeline.rst
+++ /dev/null
@@ -1,22 +0,0 @@
-.. Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- (the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
-.. http://www.apache.org/licenses/LICENSE-2.0
-
-.. Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
-
-Kyuubi Beeline
-==============
-
-.. warning:: The document you are visiting now is incomplete, please help
kyuubi community to fix it if appropriate for you.
-
-
-
diff --git
a/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/BeeLine.java
b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/BeeLine.java
index 58a35a813..3f749334e 100644
--- a/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/BeeLine.java
+++ b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/BeeLine.java
@@ -89,13 +89,7 @@ import org.apache.commons.cli.ParseException;
import org.apache.commons.io.IOUtils;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hive.beeline.common.util.ShutdownHookManager;
-import org.apache.hive.beeline.hs2connection.BeelineConfFileParseException;
-import org.apache.hive.beeline.hs2connection.BeelineSiteParseException;
-import org.apache.hive.beeline.hs2connection.BeelineSiteParser;
-import org.apache.hive.beeline.hs2connection.HS2ConnectionFileParser;
-import org.apache.hive.beeline.hs2connection.HS2ConnectionFileUtils;
-import org.apache.hive.beeline.hs2connection.HiveSiteHS2ConnectionFileParser;
-import org.apache.hive.beeline.hs2connection.UserHS2ConnectionFileParser;
+import org.apache.hive.beeline.hs2connection.*;
import org.apache.kyuubi.jdbc.hive.JdbcConnectionParams;
import org.apache.kyuubi.jdbc.hive.JdbcUriParseException;
import org.apache.kyuubi.jdbc.hive.Utils;
@@ -954,7 +948,6 @@ public class BeeLine implements Closeable {
/*
* Attempts to make a connection using default HS2 connection config file if
available
* if there connection is not made return false
- *
*/
private boolean defaultBeelineConnect(CommandLine cl) {
String url;
@@ -972,11 +965,9 @@ public class BeeLine implements Closeable {
}
private String getDefaultConnectionUrl(CommandLine cl) throws
BeelineConfFileParseException {
- Properties mergedConnectionProperties = new Properties();
JdbcConnectionParams jdbcConnectionParams = null;
BeelineSiteParser beelineSiteParser = getUserBeelineSiteParser();
UserHS2ConnectionFileParser userHS2ConnFileParser =
getUserHS2ConnFileParser();
- Properties userConnectionProperties = new Properties();
if (!userHS2ConnFileParser.configExists() &&
!beelineSiteParser.configExists()) {
// nothing to do if there is no user HS2 connection configuration file
@@ -1007,11 +998,13 @@ public class BeeLine implements Closeable {
}
}
+ Properties userConnectionProperties = new Properties();
if (userHS2ConnFileParser.configExists()) {
// get the connection properties from user specific config file
userConnectionProperties =
userHS2ConnFileParser.getConnectionProperties();
}
+ Properties mergedConnectionProperties = userConnectionProperties;
if (jdbcConnectionParams != null) {
String userName = cl.getOptionValue("n");
if (userName != null) {
@@ -1024,25 +1017,30 @@ public class BeeLine implements Closeable {
mergedConnectionProperties =
HS2ConnectionFileUtils.mergeUserConnectionPropertiesAndBeelineSite(
userConnectionProperties, jdbcConnectionParams);
- } else {
- mergedConnectionProperties = userConnectionProperties;
}
- // load the HS2 connection url properties from hive-site.xml if it is
present in the classpath
+ Properties serverConnectionProperties = new Properties();
+ KyuubiConfFileParser kyuubiConfFileParser = getKyuubiConfFileParser();
HS2ConnectionFileParser hiveSiteParser =
getHiveSiteHS2ConnectionFileParser();
- Properties hiveSiteConnectionProperties =
hiveSiteParser.getConnectionProperties();
- // add/override properties found from hive-site with user-specific
properties
+ if (kyuubiConfFileParser.configExists()) {
+ // load the Kyuubi connection url properties from kyuubi-defaults.conf
if present
+ serverConnectionProperties =
kyuubiConfFileParser.getConnectionProperties();
+ } else if (hiveSiteParser.configExists()) {
+ // load the HS2 connection url properties from hive-site.xml if present
+ serverConnectionProperties = hiveSiteParser.getConnectionProperties();
+ }
+ // add/override properties found from kyuubi-defaults.conf with
user-specific properties
for (String key : mergedConnectionProperties.stringPropertyNames()) {
- if (hiveSiteConnectionProperties.containsKey(key)) {
+ if (serverConnectionProperties.containsKey(key)) {
debug(
"Overriding connection url property "
+ key
+ " from user connection configuration file");
}
- hiveSiteConnectionProperties.setProperty(key,
mergedConnectionProperties.getProperty(key));
+ serverConnectionProperties.setProperty(key,
mergedConnectionProperties.getProperty(key));
}
// return the url based on the aggregated connection properties
- return HS2ConnectionFileUtils.getUrl(hiveSiteConnectionProperties);
+ return HS2ConnectionFileUtils.getUrl(serverConnectionProperties);
}
/*
@@ -1069,6 +1067,11 @@ public class BeeLine implements Closeable {
return new HiveSiteHS2ConnectionFileParser();
}
+ @VisibleForTesting
+ public KyuubiConfFileParser getKyuubiConfFileParser() {
+ return new KyuubiConfFileParser();
+ }
+
int runInit() {
String[] initFiles = getOpts().getInitFiles();
diff --git
a/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParseException.java
b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParseException.java
new file mode 100644
index 000000000..b62d1451c
--- /dev/null
+++
b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParseException.java
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.beeline.hs2connection;
+
+public class KyuubiConfFileParseException extends
BeelineConfFileParseException {
+ KyuubiConfFileParseException(String msg, Exception e) {
+ super(msg, e);
+ }
+
+ public KyuubiConfFileParseException(String msg) {
+ super(msg);
+ }
+}
diff --git
a/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java
b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java
new file mode 100644
index 000000000..dd25aeafc
--- /dev/null
+++
b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.hs2connection;
+
+import java.io.IOException;
+import java.io.Reader;
+import java.net.SocketException;
+import java.net.UnknownHostException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.*;
+import org.apache.kyuubi.util.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/*
+ * Looks for a kyuubi-defaults.conf from KYUUBI_CONF_DIR. If found, this class
parses
+ * the kyuubi-defaults.conf to return a set of connection properties which can
be used
+ * to construct the connection url for Beeline connection
+ */
+public class KyuubiConfFileParser implements HS2ConnectionFileParser {
+ private static final Logger LOG =
LoggerFactory.getLogger(KyuubiConfFileParser.class);
+ private static final String TRUSTSTORE_PASS_PROP =
"javax.net.ssl.trustStorePassword";
+ private static final String TRUSTSTORE_PROP = "javax.net.ssl.trustStore";
+
+ private final boolean loaded;
+ private final Properties kyuubiDefaults;
+
+ public KyuubiConfFileParser() {
+ kyuubiDefaults = new Properties();
+ Path filePath = Paths.get(System.getenv("KYUUBI_CONF_DIR"),
"kyuubi-defaults.conf");
+ if (Files.isRegularFile(filePath)) {
+ LOG.info("Using {} to construct the connection URL",
filePath.toAbsolutePath());
+ try (Reader reader = Files.newBufferedReader(filePath,
StandardCharsets.UTF_8)) {
+ kyuubiDefaults.load(reader);
+ } catch (IOException rethrow) {
+ throw new RuntimeException(rethrow);
+ }
+ loaded = true;
+ } else {
+ LOG.debug("kyuubi-defaults.conf not found for constructing the
connection URL");
+ loaded = false;
+ }
+ }
+
+ // only for testing
+ KyuubiConfFileParser(Properties mockedProperties) {
+ kyuubiDefaults = mockedProperties;
+ loaded = true;
+ }
+
+ @Override
+ public Properties getConnectionProperties() throws
KyuubiConfFileParseException {
+ Properties props = new Properties();
+ if (!configExists() || thriftMode() == THRIFT_MODE.DISABLED) {
+ return props;
+ }
+ props.setProperty(HS2ConnectionFileParser.URL_PREFIX_PROPERTY_KEY,
"jdbc:kyuubi://");
+ addHosts(props);
+ addSSL(props);
+ addKerberos(props);
+ addHttp(props);
+ return props;
+ }
+
+ // TODO: Kyuubi has different logic to handle SSL and QOP properties with
HiveServer2.
+ // We need to revise this part later
+ private void addSSL(Properties props) {
+ // if (!conf.getBoolean("hive.server2.use.SSL", false)) {
+ // return;
+ // } else {
+ // props.setProperty("ssl", "true");
+ // }
+ // String truststore = System.getenv(TRUSTSTORE_PROP);
+ // if (truststore != null && truststore.isEmpty()) {
+ // props.setProperty("sslTruststore", truststore);
+ // }
+ // String trustStorePassword = System.getenv(TRUSTSTORE_PASS_PROP);
+ // if (trustStorePassword != null && !trustStorePassword.isEmpty()) {
+ // props.setProperty("trustStorePassword", trustStorePassword);
+ // }
+ String saslQop =
kyuubiDefaults.getProperty("kyuubi.authentication.sasl.qop", "auth");
+ if (!"auth".equalsIgnoreCase(saslQop)) {
+ props.setProperty("sasl.qop", saslQop);
+ }
+ }
+
+ private void addKerberos(Properties props) {
+ if (kerberosEnabled()) {
+ props.setProperty("principal",
kyuubiDefaults.getProperty("kyuubi.kinit.principal"));
+ }
+ }
+
+ private void addHttp(Properties props) {
+ THRIFT_MODE thriftMode = thriftMode();
+ assert thriftMode == THRIFT_MODE.BINARY || thriftMode == THRIFT_MODE.HTTP;
+
+ if (thriftMode == THRIFT_MODE.HTTP) {
+ props.setProperty("transportMode", "http");
+ String path =
kyuubiDefaults.getProperty("kyuubi.frontend.thrift.http.path", "cliservice");
+ props.setProperty("httpPath", path);
+ }
+ }
+
+ private void addHosts(Properties props) throws KyuubiConfFileParseException {
+ THRIFT_MODE thriftMode = thriftMode();
+ assert thriftMode == THRIFT_MODE.BINARY || thriftMode == THRIFT_MODE.HTTP;
+
+ String host =
kyuubiDefaults.getProperty("kyuubi.frontend.advertised.host");
+ if (host == null) {
+ if (thriftMode == THRIFT_MODE.BINARY) {
+ host =
kyuubiDefaults.getProperty("kyuubi.frontend.thrift.binary.bind.host");
+ } else { // THRIFT_MODE.HTTP
+ host =
kyuubiDefaults.getProperty("kyuubi.frontend.thrift.http.bind.host");
+ }
+ }
+ if (host == null) {
+ host = kyuubiDefaults.getProperty("kyuubi.frontend.bind.host");
+ }
+
+ if (host == null) {
+ try {
+ String useHostname =
+
kyuubiDefaults.getProperty("kyuubi.frontend.connection.url.use.hostname",
"true");
+ if (Boolean.parseBoolean(useHostname)) {
+ host = JavaUtils.findLocalInetAddress().getCanonicalHostName();
+ } else {
+ host = JavaUtils.findLocalInetAddress().getHostAddress();
+ }
+ } catch (UnknownHostException | SocketException rethrow) {
+ throw new KyuubiConfFileParseException(rethrow.getMessage(), rethrow);
+ }
+ }
+
+ int portNum = getPortNum(thriftMode);
+ props.setProperty("hosts", host + ":" + portNum);
+ }
+
+ private int getPortNum(THRIFT_MODE thriftMode) {
+ assert thriftMode == THRIFT_MODE.BINARY || thriftMode == THRIFT_MODE.HTTP;
+
+ String port;
+ if (thriftMode == THRIFT_MODE.BINARY) {
+ port =
kyuubiDefaults.getProperty("kyuubi.frontend.thrift.binary.bind.port");
+ if (port == null) {
+ port = kyuubiDefaults.getProperty("kyuubi.frontend.bind.port",
"10009");
+ }
+ } else { // THRIFT_MODE.HTTP
+ port =
kyuubiDefaults.getProperty("kyuubi.frontend.thrift.http.bind.port", "10010");
+ }
+ return Integer.parseInt(port);
+ }
+
+ @Override
+ public boolean configExists() {
+ return loaded;
+ }
+
+ private enum THRIFT_MODE {
+ BINARY,
+ HTTP,
+ DISABLED
+ }
+
+ // Kyuubi supports enabling THRIFT_BINARY and THRIFT_HTTP at the same time,
in such a case,
+ // return THRIFT_MODE.BINARY
+ private THRIFT_MODE thriftMode() {
+ String[] protocols =
+ kyuubiDefaults.getProperty("kyuubi.frontend.protocols",
"THRIFT_BINARY,REST").split(",");
+ for (String protocol : protocols) {
+ if (protocol.equalsIgnoreCase("THRIFT_BINARY")) {
+ return THRIFT_MODE.BINARY;
+ } else if (protocol.equalsIgnoreCase("THRIFT_HTTP")) {
+ return THRIFT_MODE.HTTP;
+ }
+ }
+ return THRIFT_MODE.DISABLED;
+ }
+
+ private boolean kerberosEnabled() {
+ for (String auth : kyuubiDefaults.getProperty("kyuubi.authentication",
"NONE").split(",")) {
+ if (auth.equalsIgnoreCase("KERBEROS")) {
+ return true;
+ }
+ }
+ return false;
+ }
+}
diff --git
a/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
index 596f52cba..147c74ca5 100644
---
a/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
+++
b/kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/UserHS2ConnectionFileParser.java
@@ -53,6 +53,10 @@ public class UserHS2ConnectionFileParser implements
HS2ConnectionFileParser {
public UserHS2ConnectionFileParser() {
// file locations to be searched in the correct order
locations.add(DEFAULT_BEELINE_USER_CONF_LOCATION +
DEFAULT_CONNECTION_CONFIG_FILE_NAME);
+ if (System.getenv("KYUUBI_CONF_DIR") != null) {
+ locations.add(
+ System.getenv("KYUUBI_CONF_DIR") + File.separator +
DEFAULT_CONNECTION_CONFIG_FILE_NAME);
+ }
if (System.getenv("HIVE_CONF_DIR") != null) {
locations.add(
System.getenv("HIVE_CONF_DIR") + File.separator +
DEFAULT_CONNECTION_CONFIG_FILE_NAME);
@@ -76,7 +80,7 @@ public class UserHS2ConnectionFileParser implements
HS2ConnectionFileParser {
log.debug("User connection configuration file not found");
return props;
}
- log.info("Using connection configuration file at " + fileLocation);
+ log.info("Using connection configuration file at {}", fileLocation);
props.setProperty(HS2ConnectionFileParser.URL_PREFIX_PROPERTY_KEY,
"jdbc:hive2://");
// load the properties from config file
Configuration conf = new Configuration(false);
diff --git
a/kyuubi-hive-beeline/src/test/java/org/apache/hive/beeline/hs2connection/TestKyuubiConfFileParser.java
b/kyuubi-hive-beeline/src/test/java/org/apache/hive/beeline/hs2connection/TestKyuubiConfFileParser.java
new file mode 100644
index 000000000..46b10bdd2
--- /dev/null
+++
b/kyuubi-hive-beeline/src/test/java/org/apache/hive/beeline/hs2connection/TestKyuubiConfFileParser.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.hs2connection;
+
+import static org.junit.Assert.assertEquals;
+
+import java.net.SocketException;
+import java.net.UnknownHostException;
+import java.util.Properties;
+import org.apache.kyuubi.util.JavaUtils;
+import org.junit.Test;
+
+public class TestKyuubiConfFileParser {
+
+ @Test
+ public void testParseKerberosEnabled()
+ throws KyuubiConfFileParseException, SocketException,
UnknownHostException {
+ Properties kyuubiConf = new Properties();
+ kyuubiConf.put("kyuubi.authentication", "KERBEROS");
+ kyuubiConf.put("kyuubi.kinit.principal", "kyuubi/[email protected]");
+ KyuubiConfFileParser kyuubiConfFileParser = new
KyuubiConfFileParser(kyuubiConf);
+ Properties result = kyuubiConfFileParser.getConnectionProperties();
+ assertEquals(3, result.size());
+ assertEquals(
+ JavaUtils.findLocalInetAddress().getCanonicalHostName() + ":10009",
result.get("hosts"));
+ assertEquals("jdbc:kyuubi://",
result.get(HS2ConnectionFileParser.URL_PREFIX_PROPERTY_KEY));
+ assertEquals("kyuubi/[email protected]", result.get("principal"));
+ }
+}
diff --git a/kyuubi-hive-beeline/src/test/resources/kyuubi-defaults.conf
b/kyuubi-hive-beeline/src/test/resources/kyuubi-defaults.conf
new file mode 100644
index 000000000..5949a9b0b
--- /dev/null
+++ b/kyuubi-hive-beeline/src/test/resources/kyuubi-defaults.conf
@@ -0,0 +1,40 @@
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+## Kyuubi Configurations
+
+#
+# kyuubi.authentication NONE
+#
+# kyuubi.frontend.bind.host 10.0.0.1
+# kyuubi.frontend.protocols THRIFT_BINARY,REST
+# kyuubi.frontend.thrift.binary.bind.port 10009
+# kyuubi.frontend.rest.bind.port 10099
+#
+# kyuubi.engine.type SPARK_SQL
+# kyuubi.engine.share.level USER
+# kyuubi.session.engine.initialize.timeout PT3M
+#
+# kyuubi.ha.addresses zk1:2181,zk2:2181,zk3:2181
+# kyuubi.ha.namespace kyuubi
+#
+
+# Details in
https://kyuubi.readthedocs.io/en/master/configuration/settings.html
+kyuubi.engine.type=SPARK_SQL
+kyuubi.engine.share.level=USER
+kyuubi.session.engine.initialize.timeout=PT3M
\ No newline at end of file
