Hi, Abdelfatah, How to you read these files? spark.read.parquet or spark.sql? Could you show some code?
On Wed, Feb 15, 2017 at 8:47 PM, Ahmed Kamal Abdelfatah < ahmed.abdelfa...@careem.com> wrote: > Hi folks, > > > > How can I force spark sql to recursively get data stored in parquet format > from subdirectories ? In Hive, I could achieve this by setting few Hive > configs. > > > > set hive.input.dir.recursive=true; > > set hive.mapred.supports.subdirectories=true; > > set hive.supports.subdirectories=true; > > set mapred.input.dir.recursive=true; > > > > I tried to set these configs through spark sql queries but I get 0 records > all the times compared to hive which get me the expected results. I also > put these confs in hive-site.xml file but nothing changed. How can I handle > this issue ? > > > > Spark Version : 2.1.0 > > I used Hive 2.1.1 on emr-5.3.1 > > > > *Regards, * > > > > > *Ahmed Kamal* > *MTS in Data Science* > > *Email: **ahmed.abdelfa...@careem.com <ahmed.abdelfa...@careem.com>* > > > > >
