sorry for hijacking this thread.
@irving, how do you restart a spark job from checkpoint?
Regards
Sumit Chawla
On Fri, Dec 16, 2016 at 2:24 AM, Selvam Raman <sel...@gmail.com> wrote:
> Hi,
>
> Acutally my requiremnt is read the parquet file which is 100 partition.
> Then i use foreachpartition to read the data and process it.
>
> My sample code
>
> public static void main(String[] args) {
>
>
> SparkSession sparkSession = SparkSession.builder().appName("checkpoint
> verification").getOrCreate();
>
> sparkSession.implicits();
>
> sparkSession.sparkContext().setCheckpointDir("Checkpoint/Dec16");
>
> Dataset<Row> sampleData=sparkSession.read().parquet("filepath");
>
> sampleData.foreachPartition(new ForeachPartitionFunction<Row>(){
>
>
> /**
>
> *
>
> */
>
> private static final long serialVersionUID = 1L;
>
>
> @Override
>
> public void call(Iterator<Row> row) throws Exception
>
> {
>
>
> while(row.hasNext())
>
> {
>
> //Process data and insert into No-Sql DB
>
> }
>
> }
>
> });
>
> }
>
> }
>
>
>
> Now where can i apply rdd.checkpoint().
>
>
>
> Thanks,
>
> selvam
>
>
>
> On Thu, Dec 15, 2016 at 10:44 PM, Selvam Raman <sel...@gmail.com> wrote:
>
>> I am using java. I will try and let u know.
>> On Dec 15, 2016 8:45 PM, "Irving Duran" <irving.du...@gmail.com> wrote:
>>
>>> Not sure what programming language you are using, but in python you can
>>> do "sc.setCheckpointDir('~/apps/spark-2.0.1-bin-hadoop2.7/checkpoint/')".
>>> This will store checkpoints on that directory that I called checkpoint.
>>>
>>>
>>> Thank You,
>>>
>>> Irving Duran
>>>
>>> On Thu, Dec 15, 2016 at 10:33 AM, Selvam Raman <sel...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> is there any provision in spark batch for checkpoint.
>>>>
>>>> I am having huge data, it takes more than 3 hours to process all data.
>>>> I am currently having 100 partitions.
>>>>
>>>> if the job fails after two hours, lets say it has processed 70
>>>> partition. should i start spark job from the beginning or is there way for
>>>> checkpoint provision.
>>>>
>>>> Checkpoint,what i am expecting is start from 71 partition to till end.
>>>>
>>>> Please give me your suggestions.
>>>>
>>>> --
>>>> Selvam Raman
>>>> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>>>>
>>>
>>>
>
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>
