Re: Spark SQL insert overwrite table not showing all the partition.

[Bijay Kumar Pathak]

Hi Zhan,

I tried with IF NOT EXISTS clause and still I cannot see the first
partition only the partition with last insert overwrite is present in
the table.

Thanks,
Bijay

On Thu, Apr 21, 2016 at 11:18 PM, Zhan Zhang  wrote:

> INSERT OVERWRITE will overwrite any existing data in the table or partition
>
>- unless IF NOT EXISTS is provided for a partition (as of Hive 0.9.0
>).
>
>
>
> Thanks.
>
> Zhan Zhang
>
> On Apr 21, 2016, at 3:20 PM, Bijay Kumar Pathak  wrote:
>
> Hi,
>
> I have a job which writes to the Hive table with dynamic partition. Inside
> the job,  I am writing into the table two-time but I am only seeing the
> partition with last write although I can see in the Spark UI it is
> processing data fro both the partition.
>
> Below is the query I am using to write to the table.
>
> hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
>   SELECT * from temp_table
> """.format(date_val)
>  )
>
>
>
> Thanks,
> Bijay
>
>
>

Re: Spark SQL insert overwrite table not showing all the partition.

[Zhan Zhang]

INSERT OVERWRITE will overwrite any existing data in the table or partition

  *   unless IF NOT EXISTS is provided for a partition (as of Hive 
0.9.0).


Thanks.

Zhan Zhang

On Apr 21, 2016, at 3:20 PM, Bijay Kumar Pathak 
mailto:bkpat...@mtu.edu>> wrote:

Hi,

I have a job which writes to the Hive table with dynamic partition. Inside the 
job,  I am writing into the table two-time but I am only seeing the partition 
with last write although I can see in the Spark UI it is processing data fro 
both the partition.

Below is the query I am using to write to the table.

hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
  SELECT * from temp_table
""".format(date_val)
 )


Thanks,
Bijay

Spark SQL insert overwrite table not showing all the partition.

[Bijay Kumar Pathak]

Hi,

I have a job which writes to the Hive table with dynamic partition. Inside
the job,  I am writing into the table two-time but I am only seeing the
partition with last write although I can see in the Spark UI it is
processing data fro both the partition.

Below is the query I am using to write to the table.

hive_c.sql("""INSERT OVERWRITE TABLE base_table PARTITION (date='{1}', date_2)
  SELECT * from temp_table
""".format(date_val)
 )



Thanks,
Bijay