RollingSink - question on a failure scenario

[vpra...@gmail.com]

Hi,

Is there a chance of data loss if there is a failure between the checkpoint
completion and when "notifyCheckpointComplete" is invoked.

The pending files are moved to final state in the "notifyCheckpointComplete"
method. So if there is a failure in this method or just before the method is
invoked the data in the pending files is lost, am I missing something ?

Thanks,
Prabhu 



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/RollingSink-question-on-a-failure-scenario-tp7735.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Failed job restart - flink on yarn

[vpra...@gmail.com]

Hi,

I have a flink streaming job that reads from kafka, performs a aggregation
in a window, it ran fine for a while however when the number of events in a
window crossed a certain limit , the yarn containers failed with Out Of
Memory. The job was running with 10G containers. 

We have about 64G memory on the machine and now I want to restart the job 
with a 20G container (we ran some tests and 20G should be good enough to
accomodate all the elements from the window).

Is there a way to restart the job from the last checkpoint ? 

When I resubmit the job, it starts from the last committed offsets however
the events that were held in the window at the time of checkpointing seem to
get lost. Is there a way to recover the events buffered within the window
and were checkpointed before the failure ?

Thanks,
Prabhu



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Failed-job-restart-flink-on-yarn-tp7764.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: Failed job restart - flink on yarn

[vpra...@gmail.com]

Hi Jamie,

Thanks for the reply.

Yeah i looked at save points, i want to start my job only from the last
checkpoint, this means I have to keep track of when the checkpoint was
taken and the trigger a save point. I am not sure this is the way to go. My
state backend is HDFS and I can see that the checkpoint path has the data
that has been buffered in the window.

I want to start the job in a way such that it will read the checkpointed
data before the failure and continue processing.

I realise that the checkpoints are used whenever there is a container
failure, and a new container is obtained. In my case the job failed because
a container failed for the maximum AllowedN umber of failures

Thanks,
Prabhu

On Fri, Jul 1, 2016 at 3:54 PM, Jamie Grier [via Apache Flink User Mailing
List archive.]  wrote:

> Hi Prabhu,
>
> Have you taken a look at Flink's savepoints feature?  This allows you to
> make snapshots of your job's state on demand and then at any time restart
> your job from that point:
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/savepoints.html
>
> Also know that you can use Flink disk-backed state backend as well if
> you're job state is larger than fits in memory.  See
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/state_backends.html#the-rocksdbstatebackend
>
>
> -Jamie
>
>
> On Fri, Jul 1, 2016 at 1:34 PM, [hidden email]
>  <[hidden email]
> > wrote:
>
>> Hi,
>>
>> I have a flink streaming job that reads from kafka, performs a aggregation
>> in a window, it ran fine for a while however when the number of events in
>> a
>> window crossed a certain limit , the yarn containers failed with Out Of
>> Memory. The job was running with 10G containers.
>>
>> We have about 64G memory on the machine and now I want to restart the job
>> with a 20G container (we ran some tests and 20G should be good enough to
>> accomodate all the elements from the window).
>>
>> Is there a way to restart the job from the last checkpoint ?
>>
>> When I resubmit the job, it starts from the last committed offsets however
>> the events that were held in the window at the time of checkpointing seem
>> to
>> get lost. Is there a way to recover the events buffered within the window
>> and were checkpointed before the failure ?
>>
>> Thanks,
>> Prabhu
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Failed-job-restart-flink-on-yarn-tp7764.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>
>
> --
>
> Jamie Grier
> data Artisans, Director of Applications Engineering
> @jamiegrier 
> [hidden email] 
>
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Failed-job-restart-flink-on-yarn-tp7764p7767.html
> To unsubscribe from Failed job restart - flink on yarn, click here
> 
> .
> NAML
> 
>




--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Failed-job-restart-flink-on-yarn-tp7764p7771.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: Failed job restart - flink on yarn

[vpra...@gmail.com]

Thanks for the reply, It would be great to have the feature to restart a
failed job from the last checkpoint.


Is there a way to pass the initial set of partition-offsets to the
kafka-client ? In that case I can maintain a list of last processed offsets
from within my window operation (possibly store the offsets in some
database) and use that to bootstrap the kafka client upon restart.

I realize that I can probably reset the offsets for the consumer group from
some external program to the last fully processed offsets and restart the
job, just want to confirm if there is already a feature in the kafka-client.

Thanks,
Prabhu

On Mon, Jul 4, 2016 at 2:17 AM, Ufuk Celebi [via Apache Flink User Mailing
List archive.]  wrote:

> If you just re-submit the job without a savepoint, the Kafka consumer
> will by default start processing from the latest offset and the
> operators will be in an empty state. It should be possible to add a
> feature to Flink, which allows turning the latest checkpoint to a
> savepoint, from which you then could resume the job after increasing
> the container memory. But I'm afraid that this won't make it to the
> next release though. I will open an issue for it though.
>
> A work around (more a hack) would be to run in HA mode
> (
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/jobmanager_high_availability.html)
>
> and just shut down the YARN containers without cancelling the job. The
> latest checkpoint meta data should be stored in ZooKeeper and resumed
> when you restart the cluster. It's really more a hack/abuse of HA
> though.
>
> – Ufuk
>
>
> On Sat, Jul 2, 2016 at 7:09 AM, [hidden email]
>  <[hidden email]
> > wrote:
>
> > Hi Jamie,
> >
> > Thanks for the reply.
> >
> > Yeah i looked at save points, i want to start my job only from the last
> > checkpoint, this means I have to keep track of when the checkpoint was
> taken
> > and the trigger a save point. I am not sure this is the way to go. My
> state
> > backend is HDFS and I can see that the checkpoint path has the data that
> has
> > been buffered in the window.
> >
> > I want to start the job in a way such that it will read the checkpointed
> > data before the failure and continue processing.
> >
> > I realise that the checkpoints are used whenever there is a container
> > failure, and a new container is obtained. In my case the job failed
> because
> > a container failed for the maximum AllowedN umber of failures
> >
> > Thanks,
> > Prabhu
> >
> > On Fri, Jul 1, 2016 at 3:54 PM, Jamie Grier [via Apache Flink User
> Mailing
> > List archive.] <[hidden email]> wrote:
> >>
> >> Hi Prabhu,
> >>
> >> Have you taken a look at Flink's savepoints feature?  This allows you
> to
> >> make snapshots of your job's state on demand and then at any time
> restart
> >> your job from that point:
> >>
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/savepoints.html
> >>
> >> Also know that you can use Flink disk-backed state backend as well if
> >> you're job state is larger than fits in memory.  See
> >>
> https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/state_backends.html#the-rocksdbstatebackend
> >>
> >>
> >> -Jamie
> >>
> >>
> >> On Fri, Jul 1, 2016 at 1:34 PM, [hidden email] <[hidden email]> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I have a flink streaming job that reads from kafka, performs a
> >>> aggregation
> >>> in a window, it ran fine for a while however when the number of events
> in
> >>> a
> >>> window crossed a certain limit , the yarn containers failed with Out
> Of
> >>> Memory. The job was running with 10G containers.
> >>>
> >>> We have about 64G memory on the machine and now I want to restart the
> job
> >>> with a 20G container (we ran some tests and 20G should be good enough
> to
> >>> accomodate all the elements from the window).
> >>>
> >>> Is there a way to restart the job from the last checkpoint ?
> >>>
> >>> When I resubmit the job, it starts from the last committed offsets
> >>> however
> >>> the events that were held in the window at the time of checkpointing
> seem
> >>> to
> >>> get lost. Is there a way to recover the events buffered within the
> window
> >>> and were checkpointed before the failure ?
> >>>
> >>> Thanks,
> >>> Prabhu
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Failed-job-restart-flink-on-yarn-tp7764.html
> >>> Sent from the Apache Flink User Mailing List archive. mailing list
> >>> archive at Nabble.com.
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Jamie Grier
> >> data Artisans, Director of Applications Engineering
> >> @jamiegrier
> >> [hidden email]
> >>
> >>
> >>
> >> 
> >> If you reply to this email, your message will be added to the
> discussion
> >> below:
> >>
> >>
> http://apache-flink-user-ma

Re: Flink kafka group question

[vpra...@gmail.com]

>From the code  in Kafka09Fetcher.java

// if checkpointing is enabled, we are not automatically committing to
Kafka.

kafkaProperties.setProperty(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,
Boolean.toString(!runtimeContext.isCheckpointingEnabled()));


If flink checkpointing is enabled, the auto commit in kafka is disabled, I
am still trying to find out how to get a external view from the kafka offset
utilities in this scenario.

Thanks,
Prabhu



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-kafka-group-question-tp8185p8374.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.

Re: Flink Kafka Consumer Behaviour

[vpra...@gmail.com]

Hi Stephan,

The flink kafka 09 connector does not do offset commits to kafka when
checkpointing is turned on. Is there a way to monitor the offset lag in this
case,

I am turning on a flink job that reads data from kafka (has about a week
data - around 7 TB) , currently the approximate way that I use is the number
of records read shown in the flink-UI and the last offset in kafka. 

Thanks,
Prabhu



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behaviour-tp8257p8375.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.