Yeah, Spark SQL Parquet support need to do some metadata discovery when
firstly importing a folder containing Parquet files, and discovered
metadata is cached.
Cheng
On 7/17/15 1:56 PM, shsh...@tsmc.com wrote:
Hi all,
our scenario is to generate lots of folders containinig parquet file and
t
Hi all,
our scenario is to generate lots of folders containinig parquet file and
then uses "add partition" to add these folder locations to a hive table;
when trying to read the hive table using Spark,
following logs would show up and took a lot of time on reading them;
but this won't happen afte