Have you seen the previous thread?
On Sat, Sep 17, 2016 at 11:34 AM, Qiang Li <q...@appannie.com> wrote:
> I ran some jobs with Spark 2.0 on Yarn, I found all tasks finished very
> quickly, but the last step, spark spend lots of time to rename or move data
> from s3 temporary directory to real directory, then I try to set
> But both doesn't work, looks like spark 2.0 removed these configs, how can
> I let spark output directly without temporary directory ?
> *This email may contain or reference confidential information and is
> intended only for the individual to whom it is addressed. Please refrain
> from distributing, disclosing or copying this email and the information
> contained within unless you are the intended recipient. If you received
> this email in error, please notify us at le...@appannie.com
> <le...@appannie.com>** immediately and remove it from your system.*