[
https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931751#comment-16931751
]
JP Bordenave edited comment on SPARK-13446 at 9/17/19 7:00 PM:
---------------------------------------------------------------
okl thanks a lot for your help
first solution, i restore all jars 1.2.1 under spark/jar folder and hdfs
/sparkjars
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema
version 1.2.0 does not match metastore's schema version 2.3.0 Metastore is not
upgraded or corrupt
at
org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6679)
i got conflict with the schema of hive 2.3.6 initialized on db mysql, when
i use schema 1.2.0 for reinit metatstore , spark-shell doesn't show error but
hive and hadoop doenst work together and fail exeption until i restore schema
2.3.6 metastore and get them back to work.
i understand why we must copy hive*2.3.6 on spark/jars and remove
hive-*1.2.1***, seem an issue with the schema
i will try second solution next week end with the patch,
i take spark 2..4.4 and hadoop 2.7.7, and like says documentation hive from
0.12.0 to 2.3.3, but this version is not more available on hive website, the
only one available is hive 2.3.6 stable vesrion
[https://www-eu.apache.org/dist/hive/]
Spark SQL is designed to be compatible with the Hive Metastore, SerDes and
UDFs. Currently, Hive SerDes and UDFs are based on Hive 1.2.1, and Spark SQL
can be connected to different versions of Hive Metastore (from 0.12.0 to 2.3.3.
Also see
[https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore]
was (Author: jpbordi):
okl thanks a lot for your help
first solution, i restore all jars 1.2.1 under spark/jar folder and hdfs
/sparkjars
Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema
version 1.2.0 does not match metastore's schema version 2.3.0 Metastore is not
upgraded or corrupt
at
org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6679)
i got conflict with the schema of hive 2.3.6 initialized on db mysql, when
i use schema 1.2.0 for reinit metatstore , spark-shell doesn't show error but
hive and hadoop doenst work together and fail exeption until i restore schema
2.3.6 metastore and get them back to work.
i understand why we must copy hive*2.3.6 on spark/jars and remove hive-*1.2.1***
i will try second solution next week end with the patch,
i take spark 2..4.4 and hadoop 2.7.7, and like says documentation hive from
0.12.0 to 2.3.3, but this version is not more available on hive website, the
only one available is hive 2.3.6 stable vesrion
[https://www-eu.apache.org/dist/hive/]
Spark SQL is designed to be compatible with the Hive Metastore, SerDes and
UDFs. Currently, Hive SerDes and UDFs are based on Hive 1.2.1, and Spark SQL
can be connected to different versions of Hive Metastore (from 0.12.0 to 2.3.3.
Also see
[https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore]
> Spark need to support reading data from Hive 2.0.0 metastore
> ------------------------------------------------------------
>
> Key: SPARK-13446
> URL: https://issues.apache.org/jira/browse/SPARK-13446
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Lifeng Wang
> Assignee: Xiao Li
> Priority: Major
> Fix For: 2.2.0
>
>
> Spark provided HIveContext class to read data from hive metastore directly.
> While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has
> released, it's better to upgrade to support Hive 2.0.0.
> {noformat}
> 16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI
> thrift://hsw-node13:9083
> 16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current
> connections: 1
> 16/02/23 02:35:02 INFO metastore: Connected to metastore.
> Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
> at
> org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
> at
> org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
> at
> org.apache.spark.sql.hive.HiveContext$$anon$1.<init>(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
> at
> org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:421)
> at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:72)
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:739)
> at org.apache.spark.sql.SQLContext.table(SQLContext.scala:735)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
