Any example code?
In pyspark:
sqlContex.sql("use mytable")
my_df.saveAsTable("tmp_spark_debug", mode="overwrite")
1. The code above seems not register the table in hive. I have to create
table from hdfs in hive, it reports some format error: rcformat and parquet.
2. Rerun the saveAsTable using mode="overwrite" in saveAsTable, it reports
the table already exists.
3. Sometimes it creates a directory in hive/warehouse/tmp_spark_debug, not
in hive/warehouse/mytable/tmp_spark_debug.
My goal is simple:
df.saveAsTable('blablabla') // create a hive table in some database, then
it can be visited by hive.
I tried lots of time, it seems there are lots of bug in pyspark. Or my
mehtod is wrong?
2016-02-21 10:04 GMT+08:00 gen tang <[email protected]>:
> Hi,
>
> You can use
> sqlContext.sql("use <your database>")
> before use dataframe.saveAsTable
>
> Hope it could be helpful
>
> Cheers
> Gen
>
>
> On Sun, Feb 21, 2016 at 9:55 AM, Glen <[email protected]> wrote:
>
>> For dataframe in spark, so the table can be visited by hive.
>>
>> --
>> Jacky Wang
>>
>
>
--
Jacky Wang
