Re: Re: spark table to hive table

JaeBoo Jung Tue, 27 May 2014 18:09:02 -0700

Title: Samsung Enterprise Portal mySingle

I already tried HiveContext as well as SqlContext.

But it seems that Spark's HiveContext is not completely same as Apache Hive.

For example, SQL like 'SELECT RANK() OVER(ORDER BY VAL1 ASC) FROM TEST LIMIT 10' works fine in Apache Hive,

but Spark's Hive Context has an error.

That's why I want to use Shark or Apache Hive in special cases.

The followings are error message on Spark.

====================================================

java.lang.RuntimeException:
Unsupported language features in query: SELECT RANK() OVER(order by val1) FROM TEST LIMIT 10
TOK_QUERY
TOK_FROM
    TOK_TABREF
      TOK_TABNAME
        TEST
TOK_INSERT
    TOK_DESTINATION
      TOK_DIR
        TOK_TMP_FILE
    TOK_SELECT
      TOK_SELEXPR
        TOK_FUNCTION
          RANK
          TOK_WINDOWSPEC
            TOK_PARTITIONINGSPEC
              TOK_ORDERBY
                TOK_TABSORTCOLNAMEASC
                  TOK_TABLE_OR_COL
                    v1
    TOK_LIMIT
      10

        at scala.sys.package$.error(package.scala:27)
        at org.apache.spark.sql.hive.HiveQl$.parseSql(HiveQl.scala:236)
        at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:81)
        at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:90)
        at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:18)
        at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:23)
        at $iwC$$iwC$$iwC$$iwC.<init>(<console>:25)
        at $iwC$$iwC$$iwC.<init>(<console>:27)
        at $iwC$$iwC.<init>(<console>:29)
        at $iwC.<init>(<console>:31)
        at <init>(<console>:33)
        at .<init>(<console>:37)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:788)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1056)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:614)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:645)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:609)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:796)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:841)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:753)
        at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:601)
        at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:608)
        at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:611)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:936)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
        at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:884)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:884)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:983)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

====================================================

------- Original Message -------

Sender : John Omernik<[email protected]>

Date : 2014-05-27 19:28 (GMT+09:00)

Title : Re: spark table to hive table

Did you try the Hive Context? Look under Hive Support here:

http://people.apache.org/~pwendell/catalyst-docs/sql-programming-guide.html

On Tue, May 27, 2014 at 2:09 AM, 정재부 <[email protected]> wrote:

Hi all,

I'm trying to compare functions available in Spark1.0 hql to original HiveQL.

But, when I tested functions such as 'rank', Spark didn't support some HiveQL functions.

In case of Shark, it supports functions as well as Hive so I want to convert parquet file, Spark SQL table to Hive Table and analyze it with Shark.

Is there any way to do this?

Thanks,

Kevin

_____________________________________________

Kevin Jung
Assistant Engineer / BDA Lab
T +82-2-6155-8349 M +82-10-9288-1984
F +82-2-6155-0251 E [email protected]

Re: Re: spark table to hive table

Reply via email to