Is your 35GB parquet file split up into multiple S3 objects or just one big
If its just one big file then I believe only one executor will be able to
work on it until some job action partitions the data into smaller chunks.
On 11 October 2016 at 06:03, Selvam Raman <sel...@gmail.com> wrote:
> I mentioned parquet as input format.
> On Oct 10, 2016 11:06 PM, "ayan guha" <guha.a...@gmail.com> wrote:
>> It really depends on the input format used.
>> On 11 Oct 2016 08:46, "Selvam Raman" <sel...@gmail.com> wrote:
>>> How spark reads data from s3 and runs parallel task.
>>> Assume I have a s3 bucket size of 35 GB( parquet file).
>>> How the sparksession will read the data and process the data parallel.
>>> How it splits the s3 data and assign to each executor task.
>>> Please share me your points.
>>> if we have RDD , then we can look at the partitions.size or length to
>>> check how many partition for a file. But how this will be accomplished in
>>> terms of S3 bucket.
>>> Selvam Raman
>>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"