This is an automated email from the ASF dual-hosted git repository. gengliang pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push: new 95df63c [SPARK-30906][SQL][TESTS][FOLLOW-UP] Set the configuration against TestHive explicitly in HiveSerDeSuite 95df63c is described below commit 95df63c45fd9c8159ddee00014023eda10df93dc Author: HyukjinKwon <gurwls...@apache.org> AuthorDate: Wed Feb 26 18:01:26 2020 -0800 [SPARK-30906][SQL][TESTS][FOLLOW-UP] Set the configuration against TestHive explicitly in HiveSerDeSuite ### What changes were proposed in this pull request? After https://github.com/apache/spark/pull/27659 (see https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/253/), the tests below fail consistently, specifically in one job https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/ in Jenkins ``` org.apache.spark.sql.hive.execution.HiveSerDeSuite.Test the default fileformat for Hive-serde tables ``` The profile is same as PR builder but seems it fails specifically in this machine. Several configurations used in `HiveSerDeSuite` are not being set presumably due to the inconsistency between `SQLConf.get` and the active Spark session described in the https://github.com/apache/spark/pull/27387, and as a side effect of the cloned session at https://github.com/apache/spark/pull/27659. This PR proposes to explicitly set the configuration against `TestHive` by using `withExistingConf` at `withSQLConf` ### Why are the changes needed? To make `spark-master-test-sbt-hadoop-2.7-hive-2.3` job pass. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Cannot reproduce in my local. Presumably it cannot be reproduced in the PR builder. We should see if the tests pass at `spark-master-test-sbt-hadoop-2.7-hive-2.3` job after this PR is merged. Closes #27705 from HyukjinKwon/SPARK-30906. Authored-by: HyukjinKwon <gurwls...@apache.org> Signed-off-by: Gengliang Wang <gengliang.w...@databricks.com> (cherry picked from commit cd3ef2249fc6bd60e4ba62ee046dcf4115864206) Signed-off-by: Gengliang Wang <gengliang.w...@databricks.com> --- .../scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala index 9a1190a..d2d3502 100644 --- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala +++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSerDeSuite.scala @@ -82,6 +82,10 @@ class HiveSerDeSuite extends HiveComparisonTest with PlanTest with BeforeAndAfte }.head } + // Make sure we set the config values to TestHive.conf. + override def withSQLConf(pairs: (String, String)*)(f: => Unit): Unit = + SQLConf.withExistingConf(TestHive.conf)(super.withSQLConf(pairs: _*)(f)) + test("Test the default fileformat for Hive-serde tables") { withSQLConf("hive.default.fileformat" -> "orc", SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT_ENABLED.key -> "true") { --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org