Re: Writing to Parquet Job turns to wait mode after even completion of job

Thank you for everyone, origin question " Every time, i write to parquet. it shows on Spark UI that stages succeeded but on spark shell it hold context on wait mode for almost 10 mins. then it clears broadcast, accumulator shared variables.". I don't think stopping context can resolve current

Re: Writing to Parquet Job turns to wait mode after even completion of job

I think you should just shut down your SparkContext at the end. sc.stop() 2016-10-21 22:47 GMT+02:00 Chetan Khatri : > Hello Spark Users, > > I am writing around 10 GB of Processed Data to Parquet where having 1 TB > of HDD and 102 GB of RAM, 16 vCore machine on Google

Re: Writing to Parquet Job turns to wait mode after even completion of job

On 24 Oct 2016, at 20:32, Cheng Lian > wrote: On 10/22/16 6:18 AM, Steve Loughran wrote: ... On Sat, Oct 22, 2016 at 3:41 AM, Cheng Lian > wrote: What version of Spark are you using and

Re: Writing to Parquet Job turns to wait mode after even completion of job

On 10/22/16 6:18 AM, Steve Loughran wrote: ... On Sat, Oct 22, 2016 at 3:41 AM, Cheng Lian > wrote: What version of Spark are you using and how many output files does the job writes out? By default, Spark versions before 1.6

Re: Writing to Parquet Job turns to wait mode after even completion of job

On 22 Oct 2016, at 00:48, Chetan Khatri > wrote: Hello Cheng, Thank you for response. I am using spark 1.6.1, i am writing around 350 gz parquet part files for single table. Processed around 180 GB of Data using Spark. Are you writing

Re: Writing to Parquet Job turns to wait mode after even completion of job

Hello Cheng, Thank you for response. I am using spark 1.6.1, i am writing around 350 gz parquet part files for single table. Processed around 180 GB of Data using Spark. On Sat, Oct 22, 2016 at 3:41 AM, Cheng Lian wrote: > What version of Spark are you using and how

Re: Writing to Parquet Job turns to wait mode after even completion of job

What version of Spark are you using and how many output files does the job writes out? By default, Spark versions before 1.6 (not including) writes Parquet summary files when committing the job. This process reads footers from all Parquet files in the destination directory and merges them