Re: write files of a specific size

[Alonso Isidoro Roman]

Check these links:

https://stackoverflow.com/questions/31610971/spark-repartition-vs-coalesce

https://medium.com/@mrpowers/managing-spark-partitions-with-coalesce-and-repartition-4050c57ad5c4



El dom., 5 may. 2019 a las 11:48, hemant singh ()
escribió:

> Based on size of the output data you can do the math of how many file you
> will need to produce 100MB files. Once you have number of files you can do
> coalesce or repartition depending on whether your job writes more or less
> output partitions.
>
> On Sun, 5 May 2019 at 2:21 PM, rajat kumar 
> wrote:
>
>> Hi All,
>> My spark sql job produces output as per default partition and creates N
>> number of files.
>> I want to create each file as 100Mb sized in the final result.
>>
>> how can I do it ?
>>
>> thanks
>> rajat
>>
>>

-- 
Alonso Isidoro Roman
[image: https://]about.me/alonso.isidoro.roman

Re: write files of a specific size

[hemant singh]

Based on size of the output data you can do the math of how many file you
will need to produce 100MB files. Once you have number of files you can do
coalesce or repartition depending on whether your job writes more or less
output partitions.

On Sun, 5 May 2019 at 2:21 PM, rajat kumar 
wrote:

> Hi All,
> My spark sql job produces output as per default partition and creates N
> number of files.
> I want to create each file as 100Mb sized in the final result.
>
> how can I do it ?
>
> thanks
> rajat
>
>