This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new e12398c [SPARK-29247][SQL] Redact sensitive information in when
construct HiveClientHive.state
e12398c is described below
commit e12398cd0536f0d041d0897ff1d5063e06d2466f
Author: angerszhu <[email protected]>
AuthorDate: Sun Sep 29 14:30:32 2019 -0700
[SPARK-29247][SQL] Redact sensitive information in when construct
HiveClientHive.state
### What changes were proposed in this pull request?
HiveClientImpl may be log sensitive information. e.g. url, secret and token:
```scala
logDebug(
s"""
|Applying Hadoop/Hive/Spark and extra properties to Hive Conf:
|$k=${if (k.toLowerCase(Locale.ROOT).contains("password")) "xxx"
else v}
""".stripMargin)
```
So redact it. Use SQLConf.get.redactOptions.
I add a new overloading function to fit this situation for one by one kv
pair situation.
### Why are the changes needed?
Redact sensitive information when construct HiveClientImpl
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
MT
Run command
` /sbin/start-thriftserver.sh`
In log we can get
```
19/09/28 08:27:02 main DEBUG HiveClientImpl:
Applying Hadoop/Hive/Spark and extra properties to Hive Conf:
hive.druid.metadata.password=*********(redacted)
```
Closes #25954 from AngersZhuuuu/SPARK-29247.
Authored-by: angerszhu <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 1d4b2f010b8e1224985f4204c3e1d88c65d71f69)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 798a608..b51eae9 100644
---
a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++
b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -52,6 +52,7 @@ import org.apache.spark.sql.execution.QueryExecutionException
import org.apache.spark.sql.execution.command.DDLUtils
import org.apache.spark.sql.hive.HiveExternalCatalog.{DATASOURCE_SCHEMA,
DATASOURCE_SCHEMA_NUMPARTS, DATASOURCE_SCHEMA_PART_PREFIX}
import org.apache.spark.sql.hive.client.HiveClientImpl._
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.util.{CircularBuffer, Utils}
@@ -167,14 +168,15 @@ private[hive] class HiveClientImpl(
// has hive-site.xml. So, HiveConf will use that to override its default
values.
// 2: we set all spark confs to this hiveConf.
// 3: we set all entries in config to this hiveConf.
- (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue)
- ++ sparkConf.getAll.toMap ++ extraConfig).foreach { case (k, v) =>
+ val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey ->
kv.getValue) ++
+ sparkConf.getAll.toMap ++ extraConfig).toMap
+ confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+ SQLConf.get.redactOptions(confMap).foreach { case (k, v) =>
logDebug(
s"""
|Applying Hadoop/Hive/Spark and extra properties to Hive Conf:
- |$k=${if (k.toLowerCase(Locale.ROOT).contains("password")) "xxx"
else v}
+ |$k=$v
""".stripMargin)
- hiveConf.set(k, v)
}
val state = new SessionState(hiveConf)
if (clientLoader.cachedHive != null) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
