hiveContext.sql() - Join query fails silently

[Jins George]

Hi All,

I have used a  hiveContext.sql() to join a temporary table created from 
Dataframe and parquet tables created in Hive.


The join query runs fine for few hours and then suddenly fails to do the 
Join. Once the issue happens the dataframe returned from 
hiveContext.sql() is empty. If I restart the job, things starts working 
again.


If anyone has faced this type of issue, please suggest

I am using spark 1.5.1, single node in local mode and hive Hive 1.1.0

Thanks,
Jins George

Re: Unable to run spark SQL Join query.

[Jins George]

Column 'itemId' is not present in table 
'success_events.sojsuccessevents1' or  'dw_bid'


did you mean  'sojsuccessevents2_spark' table  in your select query ?

Thanks,
Jins
On 01/03/2016 07:22 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) wrote:

Code:

val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)

hiveContext.sql("drop table sojsuccessevents2_spark")

hiveContext.sql("CREATE TABLE `sojsuccessevents2_spark`( `guid` string 
COMMENT 'from deserializer', `sessionkey` bigint COMMENT 'from 
deserializer', `sessionstartdate` string COMMENT 'from deserializer', 
`sojdatadate` string COMMENT 'from deserializer', `seqnum` int COMMENT 
'from deserializer', `eventtimestamp` string COMMENT 'from 
deserializer', `siteid` int COMMENT 'from deserializer', 
`successeventtype` string COMMENT 'from deserializer', `sourcetype` 
string COMMENT 'from deserializer', `itemid` bigint COMMENT 'from 
deserializer', `shopcartid` bigint COMMENT 'from deserializer', 
`transactionid` bigint COMMENT 'from deserializer', `offerid` bigint 
COMMENT 'from deserializer', `userid` bigint COMMENT 'from 
deserializer', `priorpage1seqnum` int COMMENT 'from deserializer', 
`priorpage1pageid` int COMMENT 'from deserializer', 
`exclwmsearchattemptseqnum` int COMMENT 'from deserializer', 
`exclpriorsearchpageid` int COMMENT 'from deserializer', 
`exclpriorsearchseqnum` int COMMENT 'from deserializer', 
`exclpriorsearchcategory` int COMMENT 'from deserializer', 
`exclpriorsearchl1` int COMMENT 'from deserializer', 
`exclpriorsearchl2` int COMMENT 'from deserializer', 
`currentimpressionid` bigint COMMENT 'from deserializer', 
`sourceimpressionid` bigint COMMENT 'from deserializer', 
`exclpriorsearchsqr` string COMMENT 'from deserializer', 
`exclpriorsearchsort` string COMMENT 'from deserializer', 
`isduplicate` int COMMENT 'from deserializer', `transactiondate` 
string COMMENT 'from deserializer', `auctiontypecode` int COMMENT 
'from deserializer', `isbin` int COMMENT 'from deserializer', 
`leafcategoryid` int COMMENT 'from deserializer', `itemsiteid` int 
COMMENT 'from deserializer', `bidquantity` int COMMENT 'from 
deserializer', `bidamtusd` double COMMENT 'from deserializer', 
`offerquantity` int COMMENT 'from deserializer', `offeramountusd` 
double COMMENT 'from deserializer', `offercreatedate` string COMMENT 
'from deserializer', `buyersegment` string COMMENT 'from 
deserializer', `buyercountryid` int COMMENT 'from deserializer', 
`sellerid` bigint COMMENT 'from deserializer', `sellercountryid` int 
COMMENT 'from deserializer', `sellerstdlevel` string COMMENT 'from 
deserializer', `csssellerlevel` string COMMENT 'from deserializer', 
`experimentchannel` int COMMENT 'from deserializer') ROW FORMAT SERDE 
'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' 
OUTPUTFORMAT 
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION 
'hdfs://apollo-phx-nn.vip.ebay.com:8020/user/dvasthimal/spark/successeventstaging/sojsuccessevents2 
' 
TBLPROPERTIES (