Hi, Good try. As you can see, when you run upgrade using schematool, there is a duplicate column error. Can you please look the script generated and edit to avoid duplicate column?
Not sure why the Hive guys made it complicated, I did face same issues like you. Can anyone else give a clean and better option? Best, Ravion On Dec 10, 2017 6:17 AM, "Alejandro Reina" <alex.k...@gmail.com> wrote: > I have tried what you propose, added the property to hive-site.xml, and > although with this option I can run hive, this does not solve my problem. > I'm sorry if maybe you explain me wrongly. > > I need to save a dataframe transformed into spark in hive, with the version > of scheme 2.1.1 of hive (last stable version of hive until 2 months ago), > by > default spark internally uses version 1.2.1 and this generates conflict > with > my other applications the ecosystem. > > This hiveTable created from spark, is not visible from any other > application > due to its version of the metadata, so I need to create from spark a hive > table in its corresponding version. > > Hive schematool > root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -info --verbose > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j- > log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Metastore connection URL: jdbc:mysql://localhost/ > metastore?useSSL=false > Metastore Connection Driver : com.mysql.jdbc.Driver > Metastore connection User: hive > Hive distribution version: 2.1.0 > Metastore schema version: 1.2.0 > org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema > version > is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0 > org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema > version > is not compatible. Hive Version: 2.1.0, Database Schema Version: 1.2.0 > at > org.apache.hive.beeline.HiveSchemaTool.assertCompatibleVersion( > HiveSchemaTool.java:202) > at org.apache.hive.beeline.HiveSchemaTool.showInfo( > HiveSchemaTool.java:139) > at org.apache.hive.beeline.HiveSchemaTool.main( > HiveSchemaTool.java:498) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.util.RunJar.run(RunJar.java:221) > at org.apache.hadoop.util.RunJar.main(RunJar.java:136) > *** schemaTool failed *** > > If i try upgrade schema too have error. > root@alex-bi:/usr/local/hive/bin# schematool -dbType mysql -upgradeSchema > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j- > log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type > [org.apache.logging.slf4j.Log4jLoggerFactory] > Metastore connection URL: jdbc:mysql://localhost/ > metastore?useSSL=false > Metastore Connection Driver : com.mysql.jdbc.Driver > Metastore connection User: hive > Starting upgrade metastore schema from version 1.2.0 to 2.1.0 > Upgrade script upgrade-1.2.0-to-2.0.0.mysql.sql > Error: Duplicate column name 'CQ_HIGHEST_TXN_ID' (state=42S21,code=1060) > org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! > Metastore state would be inconsistent !! > Underlying cause: java.io.IOException : Schema script failed, errorcode 2 > Use --verbose for detailed stacktrace. > *** schemaTool failed *** > > thank you very much for your time dedicated to helping me > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
