The files are processed in the order the file last modified timestamp. The path and partitioning scheme are not used for ordering.
On Thu, Jun 14, 2018 at 6:59 AM, karthikjay <[email protected]> wrote: > My parquet files are first partitioned by environment and then by date > like: > > env=testing/ > date=2018-03-04/ > part1.parquet > part2.parquet > part3.parquet > date=2018-03-05/ > part1.parquet > part2.parquet > part3.parquet > date=2018-03-06/ > part1.parquet > part2.parquet > part3.parquet > In our read stream, I do the following: > > val tunerParquetDF = spark > .readStream > .schema(...) > .format("parquet") > .option("basePath", basePath) > .option("path", basePath+"/env*") > .option("maxFilesPerTrigger", 5) > .load() > > The expected behavior is that readstream will read files in order of the > dates but the observed behavior is that files are shuffled in random order. > How do I force the date order of read in Parquet files ? > > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
