This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new 66af3b739e8 [SPARK-44941][SQL][TESTS] Turn off hive.conf.validation in
tests
66af3b739e8 is described below
commit 66af3b739e89785ec6d353ea678ae48d4d8e6389
Author: Kent Yao <[email protected]>
AuthorDate: Thu Aug 24 23:22:56 2023 +0800
[SPARK-44941][SQL][TESTS] Turn off hive.conf.validation in tests
### What changes were proposed in this pull request?
This PR turns off hive.conf.validation in tests to remove the noisy
HiveConf logs for removed Hive ConfVars. For example, a single test in
hive.SQLQuerySuite repeats four times.
```java
[info] SQLQuerySuite:
14:49:45.855 WARN org.apache.spark.sql.internal.WithTestConf$$anon$4: The
SQL config 'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and
may be removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault'
to false instead.
14:49:45.983 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.jdbc.timeout does not exist
14:49:45.983 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.retries.wait does not exist
14:49:47.651 WARN org.apache.hadoop.hive.metastore.ObjectStore: Version
information not found in metastore. hive.metastore.schema.verification is not
enabled so recording the schema version 2.3.0
14:49:47.651 WARN org.apache.hadoop.hive.metastore.ObjectStore:
setMetaStoreSchemaVersion called but recording version is disabled: version =
2.3.0, comment = Set by MetaStore hzyaoqin127.0.0.1
14:49:47.656 WARN org.apache.hadoop.hive.metastore.ObjectStore: Failed to
get database default, returning NoSuchObjectException
14:49:47.922 WARN org.apache.hadoop.hive.ql.session.SessionState:
METASTORE_FILTER_HOOK will be ignored, since
hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
14:49:47.998 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.internal.ss.authz.settings.applied.marker does not exist
14:49:47.998 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.jdbc.timeout does not exist
14:49:47.998 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.retries.wait does not exist
14:49:48.003 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location:
file:/Users/hzyaoqin/spark/sql/hive/target/tmp/hive_execution_test_group/warehouse-cd8178a1-6b68-426f-aa19-d47d211df0c9/explodetest
specified for non-external table:explodetest
14:49:49.264 WARN org.apache.hadoop.hive.metastore.ObjectStore: Failed to
get database global_temp, returning NoSuchObjectException
14:49:49.408 WARN org.apache.spark.sql.internal.SQLConf: The SQL config
'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and may be
removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault' to false
instead.
14:49:49.808 WARN org.apache.spark.sql.internal.WithTestConf$$anon$4: The
SQL config 'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and
may be removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault'
to false instead.
[info] - logical.Project should not be resolved if it contains aggregates
or generators (6 seconds, 33 milliseconds)
14:49:49.997 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.internal.ss.authz.settings.applied.marker does not exist
14:49:49.997 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.jdbc.timeout does not exist
14:49:49.997 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.retries.wait does not exist
14:49:50.004 WARN org.apache.hadoop.hive.ql.session.SessionState:
METASTORE_FILTER_HOOK will be ignored, since
hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
14:49:50.113 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.internal.ss.authz.settings.applied.marker does not exist
14:49:50.113 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.jdbc.timeout does not exist
14:49:50.113 WARN org.apache.hadoop.hive.conf.HiveConf: HiveConf of name
hive.stats.retries.wait does not exist
```
### Why are the changes needed?
The matcher for `org.apache.hadoop.hive.conf.HiveConf: HiveConf of name *
does not exist` exceeds 5000 lines. It can be saved.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
run a simple hive test locally and see the log gone.
```
[info] SQLQuerySuite:
14:57:13.274 WARN org.apache.spark.sql.internal.WithTestConf$$anon$4: The
SQL config 'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and
may be removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault'
to false instead.
14:57:15.155 WARN org.apache.hadoop.hive.metastore.ObjectStore: Version
information not found in metastore. hive.metastore.schema.verification is not
enabled so recording the schema version 2.3.0
14:57:15.155 WARN org.apache.hadoop.hive.metastore.ObjectStore:
setMetaStoreSchemaVersion called but recording version is disabled: version =
2.3.0, comment = Set by MetaStore hzyaoqin127.0.0.1
14:57:15.162 WARN org.apache.hadoop.hive.metastore.ObjectStore: Failed to
get database default, returning NoSuchObjectException
14:57:15.419 WARN org.apache.hadoop.hive.ql.session.SessionState:
METASTORE_FILTER_HOOK will be ignored, since
hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
14:57:15.499 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location:
file:/Users/hzyaoqin/spark/sql/hive/target/tmp/hive_execution_test_group/warehouse-06b4c2ad-728d-4460-ada3-bc1f80b7be0b/explodetest
specified for non-external table:explodetest
14:57:16.664 WARN org.apache.hadoop.hive.metastore.ObjectStore: Failed to
get database global_temp, returning NoSuchObjectException
14:57:16.809 WARN org.apache.spark.sql.internal.SQLConf: The SQL config
'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and may be
removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault' to false
instead.
14:57:17.199 WARN org.apache.spark.sql.internal.WithTestConf$$anon$4: The
SQL config 'spark.sql.hive.convertCTAS' has been deprecated in Spark v3.1 and
may be removed in the future. Set 'spark.sql.legacy.createHiveTableByDefault'
to false instead.
[info] - logical.Project should not be resolved if it contains aggregates
or generators (5 seconds, 953 milliseconds)
14:57:17.393 WARN org.apache.hadoop.hive.ql.session.SessionState:
METASTORE_FILTER_HOOK will be ignored, since
hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
[info] Run completed in 9 seconds, 203 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 140 s (02:20), completed 2023-8-24 14:57:17
```
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #42647 from yaooqinn/SPARK-44941.
Authored-by: Kent Yao <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
(cherry picked from commit c6d4912341fdd70a145c5cf0d7fd1bfc36ede092)
Signed-off-by: Kent Yao <[email protected]>
---
pom.xml | 1 +
project/SparkBuild.scala | 1 +
2 files changed, 2 insertions(+)
diff --git a/pom.xml b/pom.xml
index e550e16553d..f94b3b729c2 100644
--- a/pom.xml
+++ b/pom.xml
@@ -2999,6 +2999,7 @@
<spark.hadoop.hadoop.security.key.provider.path>test:///</spark.hadoop.hadoop.security.key.provider.path>
<!-- Needed by sql/hive tests. -->
<test.src.tables>src</test.src.tables>
+ <hive.conf.validation>false</hive.conf.validation>
</systemPropertyVariables>
<failIfNoTests>false</failIfNoTests>
<failIfNoSpecifiedTests>false</failIfNoSpecifiedTests>
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 331ebe0c05a..4b6f617e68f 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1593,6 +1593,7 @@ object TestSettings {
(Test / javaOptions) += "-Dspark.ui.showConsoleProgress=false",
(Test / javaOptions) += "-Dspark.unsafe.exceptionOnMemoryLeak=true",
(Test / javaOptions) +=
"-Dspark.hadoop.hadoop.security.key.provider.path=test:///",
+ (Test / javaOptions) += "-Dhive.conf.validation=false",
(Test / javaOptions) += "-Dsun.io.serialization.extendedDebugInfo=false",
(Test / javaOptions) += "-Dderby.system.durability=test",
(Test / javaOptions) += "-Dio.netty.tryReflectionSetAccessible=true",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
