The suggestion is to check: 1. Used format for write 2. Used parallelism
On Thu, Apr 14, 2022 at 7:13 PM Anil Dasari <adas...@guidewire.com> wrote: > Hello, > > > > We are upgrading spark from 2.4.7 to 3.0.1. we use spark sql (hive) to > checkpoint data frames (intermediate data). DF write is very slow in 3.0.1 > compared to 2.4.7. > > Have read the release notes and there were no major changes except managed > tables and adaptive scheduling. We are not using adaptive scheduling and > going with default config. We made changes to handle managed tables by > adding explicit paths during writes and delete. > > > > Do you have any suggestions to debug and fix the slowness problem ? > > > > Thanks, > > >
