Hi,
Acutally my requiremnt is read the parquet file which is 100 partition.
Then i use foreachpartition to read the data and process it.
My sample code
public static void main(String[] args) {
SparkSession sparkSession = SparkSession.builder().appName("checkpoint
verification").getOrCreate();
sparkSession.implicits();
sparkSession.sparkContext().setCheckpointDir("Checkpoint/Dec16");
Dataset<Row> sampleData=sparkSession.read().parquet("filepath");
sampleData.foreachPartition(new ForeachPartitionFunction<Row>(){
/**
*
*/
private static final long serialVersionUID = 1L;
@Override
public void call(Iterator<Row> row) throws Exception
{
while(row.hasNext())
{
//Process data and insert into No-Sql DB
}
}
});
}
}
Now where can i apply rdd.checkpoint().
Thanks,
selvam
On Thu, Dec 15, 2016 at 10:44 PM, Selvam Raman <[email protected]> wrote:
> I am using java. I will try and let u know.
> On Dec 15, 2016 8:45 PM, "Irving Duran" <[email protected]> wrote:
>
>> Not sure what programming language you are using, but in python you can
>> do "sc.setCheckpointDir('~/apps/spark-2.0.1-bin-hadoop2.7/checkpoint/')".
>> This will store checkpoints on that directory that I called checkpoint.
>>
>>
>> Thank You,
>>
>> Irving Duran
>>
>> On Thu, Dec 15, 2016 at 10:33 AM, Selvam Raman <[email protected]> wrote:
>>
>>> Hi,
>>>
>>> is there any provision in spark batch for checkpoint.
>>>
>>> I am having huge data, it takes more than 3 hours to process all data. I
>>> am currently having 100 partitions.
>>>
>>> if the job fails after two hours, lets say it has processed 70
>>> partition. should i start spark job from the beginning or is there way for
>>> checkpoint provision.
>>>
>>> Checkpoint,what i am expecting is start from 71 partition to till end.
>>>
>>> Please give me your suggestions.
>>>
>>> --
>>> Selvam Raman
>>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>>>
>>
>>
--
Selvam Raman
"லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"