How to set the file size for parquet Part

2015-05-19 Thread Richard Grossman
Hi I'm using spark 1.3.1 and now I can't set the size of the part generated file for parquet. The size is only 512Kb it's really to small I must made them bigger. How can set this ? Thanks

Write Parquet File with spark-streaming with Spark 1.3

2015-03-26 Thread Richard Grossman
Hi I've succeed to write kafka stream to parquet file in Spark 1.2 but I can't make it with spark 1.3 As in streaming I can't use saveAsParquetFile() because I can't add data to an existing parquet File I know that it's possible to stream data directly into parquet could you help me by providing