Seems like a good idea in general. Do other systems have similar concepts? In general it'd be easier if we can follow existing convention if there is any.
On Wed, Jul 25, 2018 at 11:50 AM John Zhuge <jzh...@apache.org> wrote: > Hi all, > > Many Spark users in my company are asking for a way to control the number > of output files in Spark SQL. There are use cases to either reduce or > increase the number. The users prefer not to use function *repartition*(n) > or *coalesce*(n, shuffle) that require them to write and deploy > Scala/Java/Python code. > > Could we introduce a query hint for this purpose (similar to Broadcast > Join Hints)? > > /*+ *COALESCE*(n, shuffle) */ > > In general, is query hint is the best way to bring DF functionality to SQL > without extending SQL syntax? Any suggestion is highly appreciated. > > This requirement is not the same as SPARK-6221 that asked for auto-merging > output files. > > Thanks, > John Zhuge >
