GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/7868
[SQL] [SPARK-8064] Fixes HiveSparkSubmitSuite for Hive 1.2.1 upgrade
This is a fork of PR #7191 authored by @steveloughran. This PR fixes
`HiveSparkSubmitSuite` test failures.
Hive 1.2.1 add time suffixes (`s`, `ms`, and `d`, etc.) to the original
default values of a bunch of `HiveConf` options that define various timeouts.
This fails `HiveContext` initialization when we are connecting to a lower
version of Hive (say 0.12.0 in `HiveSparkSubmitSuite`), because Hive 0.12.0
assumes these options should have integral values.
This PR manually filters out all such options and converts their values to
corresponding `Long` values according to their output time unit before passing
them to the metastore Hive client of lower versions.
Please refer to [this failed Jenkins build] [1] for more details about the
failed test case.
[1]:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39373/testReport/org.apache.spark.sql.hive/HiveSparkSubmitSuite/SPARK_8020__set_sql_conf_in_spark_conf/
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark
hive-1.2.1/hiveconf-timevar-fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/7868.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #7868
----
commit 4890b9d8ed2c2c6d946a782604135e0fdde652f1
Author: Steve Loughran <[email protected]>
Date: 2015-07-02T14:29:23Z
SPARK-8064, build against Hive 1.2.1
commit f61a675a96890310e06d5c22a606221beab56473
Author: Steve Loughran <[email protected]>
Date: 2015-07-15T17:31:09Z
SPARK-8064 thrift server switched to Hive 1.2.1, though it doesn't compile
everywhere
commit 6c310b4a0e3ca99c3da48dfd88a1718ef6c1d335
Author: Steve Loughran <[email protected]>
Date: 2015-07-16T12:19:57Z
SPARK-8064 subclass Hive ServerOptionsProcessor to make it public again
commit f7aa9cb155d068ec5a54709bc66dba1f07eeda1e
Author: Steve Loughran <[email protected]>
Date: 2015-07-16T20:38:23Z
SPARK-8064 everything compiles with some commenting and moving of classes
into a hive package
commit cb4f142ebf1b0839461103fb7e76d47d0c9e0712
Author: Steve Loughran <[email protected]>
Date: 2015-07-17T14:19:09Z
SPARK-8054 cut pentaho dependency from calcite
commit 00e50d68887eff15a7aaff55ea0d9a75d5a968dd
Author: Steve Loughran <[email protected]>
Date: 2015-07-22T20:23:48Z
SPARK-8064 stop excluding hive shims from dependency (commented out , for
now)
commit 27e8370d2247f8e5bb020fcfa79f6c7b6366b143
Author: Steve Loughran <[email protected]>
Date: 2015-07-22T20:24:10Z
SPARK-8064 fix some style & IDE warnings
commit c80979d4f3fa272c710095bcd01d85f89e3f1a7e
Author: Steve Loughran <[email protected]>
Date: 2015-07-22T20:47:56Z
SPARK-8064: SparkSQLCLIDriver drops remote mode support. CLISuite Tests
pass instead of timing out: undetected regression?
commit 3ebc2798c56e0c4381152d79716a465f2076e5d2
Author: Steve Loughran <[email protected]>
Date: 2015-07-22T21:43:10Z
SPARK-8064 move strings used to check for http/bin thrift services up into
constants
commit 75733fcdcdb140ddb9f643082e0d94693358565d
Author: Steve Loughran <[email protected]>
Date: 2015-07-22T21:46:10Z
SPARK-8064 change thrift binary startup message to "Starting
ThriftBinaryCLIService on port"
commit 1d59100de4336a8944f983bfc8fffb460884837a
Author: Steve Loughran <[email protected]>
Date: 2015-07-23T18:18:19Z
SPARK-8064 (unsuccessful) attempt to get rid of pentaho as a transitive
dependency of hive-exec
commit 2531099481f7cccdf3a1f3bdbbe2f3c9cf439dbd
Author: Steve Loughran <[email protected]>
Date: 2015-07-23T18:59:08Z
SPARK-8064 successful attempt to get rid of pentaho as a transitive
dependency of hive-exec
commit 463a6705ad994ea4dc4e5ebf10b8ed0f20cacb99
Author: Steve Loughran <[email protected]>
Date: 2015-07-23T19:00:01Z
SPARK-8064 run-tests.py adds a hadoop-2.6 profile, and changes info
messages to say "w/Hive 1.2.1" in console output
commit dec12cb3c757626f852e1dde7665ebb61d1fae92
Author: Steve Loughran <[email protected]>
Date: 2015-07-23T19:01:53Z
SPARK-8064: when a CLI suite test fails include the full output text in the
raised exception; this ensures that the stdout/stderr is included in jenkins
reports, so it becomes possible to diagnose the cause.
commit 8cb09c43f71e67f7dc6f24a5a937862002136657
Author: Steve Loughran <[email protected]>
Date: 2015-07-23T21:33:54Z
SPARK-8064: test resilience/assertion improvements. Independent of the rest
of the work; can be backported to earlier versions
commit 1126e5a2bf768901f79e534bcda0405e15163c38
Author: Steve Loughran <[email protected]>
Date: 2015-07-23T21:34:53Z
SPARK-8064: name of unrecognized file format wasn't appearing in error text
commit d0360f610f97d0847bb0b8ec9c3f286f78c769cd
Author: Steve Loughran <[email protected]>
Date: 2015-07-24T03:08:22Z
SPARK-8064: delicate merge in of the branch vanzin/hive-1.1
commit 3d64523ad289e3173052f34e785b6ae3a3db6a58
Author: Steve Loughran <[email protected]>
Date: 2015-07-24T16:39:55Z
SPARK-8064 improve diagns on uknown token; fix scalastyle failure
commit 26eef1ca89104441681e4b3ceb06e49dd33a5eb9
Author: Steve Loughran <[email protected]>
Date: 2015-07-24T17:12:33Z
SPARK-8064: HIVE-9039 renamed TOK_UNION => TOK_UNIONALL while adding
TOK_UNIONDISTINCT
commit d14d5ea2f942e4af722f9c0761e131b11973877a
Author: Steve Loughran <[email protected]>
Date: 2015-07-24T17:19:50Z
SPARK-8064: DATE is now a predicate; you can't use it as a field in select
ops
commit 13abaf164834226a3cbf0ae280fe09cc79ecdb3c
Author: Steve Loughran <[email protected]>
Date: 2015-07-24T19:50:05Z
SPARK-8064 Hive compatibilty tests sin sync with explain/show output from
Hive 1.2.1
commit 0b0f73892fc7015d7c2644299588c2cd7c12d311
Author: Steve Loughran <[email protected]>
Date: 2015-07-24T21:52:49Z
SPARK-8064: thrift server startup to fail fast on any exception in the main
thread
commit c829b8f7ad80bbc4a36465b62a7cbb7465705de6
Author: Steve Loughran <[email protected]>
Date: 2015-07-24T21:53:12Z
SPARK-8064: reinstate yarn-rm-server dependencies to hive-exec to ensure
that jersey server is on classpath on hadoop versions < 2.6
commit bf3a2495514c360d737b78819c3592af5736ae83
Author: Steve Loughran <[email protected]>
Date: 2015-07-27T05:22:43Z
SPARK-8064: more resubmit than fix; tighten startup timeout to 60s. Still
no obvious reason why jersey server code in spark-assembly isn't being picked
up -it hasn't been shaded
commit 1ab9bc40d84de730d2ff17e7a1a5e200203a6198
Author: Steve Loughran <[email protected]>
Date: 2015-07-27T17:36:08Z
SPARK-8064 TestHive to use sered2.thrift.test.Complex
commit 2bc29a47ab9ab5e3ce6bbe2a9ca679c42f8df979
Author: Steve Loughran <[email protected]>
Date: 2015-07-27T17:37:13Z
SPARK-8064 ParquetSuites to escape `date` field name
commit 41d6479cfd813e12cd3c81ece5b63911723e7ab6
Author: Steve Loughran <[email protected]>
Date: 2015-07-27T18:01:12Z
SPARK-8064 wrap tests with withTable() calls to avoid table-exists
exceptions
commit bca55e5ddfbccf53f30e2898ebc8f2e79346365f
Author: Steve Loughran <[email protected]>
Date: 2015-07-27T20:57:10Z
SPARK-8064 missed one of the `date` escapes
commit 3ed872f96cb988ca2fdd08a5fb3c757336a90f35
Author: Steve Loughran <[email protected]>
Date: 2015-07-27T20:57:42Z
SPARK-8064 rename field double to dbl
commit 335357f16864a6c267407529eca97b3bb18b6651
Author: Steve Loughran <[email protected]>
Date: 2015-07-27T23:19:47Z
SPARK-8064 fail fast on thrive process spawning tests on exit codes and/or
error string patterns seen in log.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
