Query data in subdirectories in Hive Partitions using Spark SQL

[Ahmed Kamal Abdelfatah]

Hi folks,

How can I force spark sql to recursively get data stored in parquet format from 
subdirectories ?  In Hive, I could achieve this by setting few Hive configs.

set hive.input.dir.recursive=true;
set hive.mapred.supports.subdirectories=true;
set hive.supports.subdirectories=true;
set mapred.input.dir.recursive=true;

I tried to set these configs through spark sql queries but I get 0 records all 
the times compared to hive which get me the expected results. I also put these 
confs in hive-site.xml file but nothing changed. How can I handle this issue ?

Spark Version : 2.1.0
I used Hive 2.1.1  on emr-5.3.1

Regards,


Ahmed Kamal
MTS in Data Science
Email: ahmed.abdelfa...@careem.com

unsubscribe

[Ahmed Kamal Abdelfatah]

-- 










*This email, including any information it contains and any 
attachments to it, is confidential and may be privileged. This email is 
intended only for the use of the named recipient(s). If you are not a named 
recipient, please notify the sender immediately by replying to this message 
and delete the original message. You should not disclose or copy this 
email, any of its contents or any attachments to it. This email may have 
been transmitted over an unsecure public network and, therefore, Careem 
does not accept responsibility for its contents or for any damage sustained 
as a result of viewing its contents or in connection with its transmission. 
Careem reserves the right to monitor all communications from or to this 
account.*