I am doing a simple count like: sqlContext.read.parquet("path").count
I have only 5000 parquet files. But generate over 20000 tasks. Each parquet file is converted from one gz text file. Please give some advice. Thanks
I am doing a simple count like: sqlContext.read.parquet("path").count
I have only 5000 parquet files. But generate over 20000 tasks. Each parquet file is converted from one gz text file. Please give some advice. Thanks