[jira] [Commented] (FLINK-3404) Extend Kafka consumers with interface StoppableFunction

2016-11-23 Thread static-max (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15690734#comment-15690734
 ] 

static-max commented on FLINK-3404:
---

I'm in need of a _stop()_ method too, I will propose an PR for this issue if 
there is interest.

> Extend Kafka consumers with interface StoppableFunction
> ---
>
> Key: FLINK-3404
> URL: https://issues.apache.org/jira/browse/FLINK-3404
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Matthias J. Sax
>
> Kafka consumers are not stoppable right now. To make them stoppable, they 
> must implement {{StoppableFunction}}. Implementing method {{stop()}} must 
> ensure, that the consumer stops pulling new messages from Kafka and issues a 
> final checkpoint with the last offset. Afterwards, {{run()}} must return.
> When implementing this, keep in mind, that the gathered checkpoint might 
> later be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3404) Extend Kafka consumers with interface StoppableFunction

2016-05-03 Thread Aljoscha Krettek (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268558#comment-15268558
 ] 

Aljoscha Krettek commented on FLINK-3404:
-

In some way, yes.

> Extend Kafka consumers with interface StoppableFunction
> ---
>
> Key: FLINK-3404
> URL: https://issues.apache.org/jira/browse/FLINK-3404
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Matthias J. Sax
>
> Kafka consumers are not stoppable right now. To make them stoppable, they 
> must implement {{StoppableFunction}}. Implementing method {{stop()}} must 
> ensure, that the consumer stops pulling new messages from Kafka and issues a 
> final checkpoint with the last offset. Afterwards, {{run()}} must return.
> When implementing this, keep in mind, that the gathered checkpoint might 
> later be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3404) Extend Kafka consumers with interface StoppableFunction

2016-05-03 Thread Martin Liesenberg (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15268522#comment-15268522
 ] 

Martin Liesenberg commented on FLINK-3404:
--

ok, I didn't realize that needed to be done first. 

and just out of curiosity, what it boils down to, is that some version of 
distributed consensus needs to be implemented here first?

> Extend Kafka consumers with interface StoppableFunction
> ---
>
> Key: FLINK-3404
> URL: https://issues.apache.org/jira/browse/FLINK-3404
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Matthias J. Sax
>
> Kafka consumers are not stoppable right now. To make them stoppable, they 
> must implement {{StoppableFunction}}. Implementing method {{stop()}} must 
> ensure, that the consumer stops pulling new messages from Kafka and issues a 
> final checkpoint with the last offset. Afterwards, {{run()}} must return.
> When implementing this, keep in mind, that the gathered checkpoint might 
> later be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3404) Extend Kafka consumers with interface StoppableFunction

2016-04-30 Thread Aljoscha Krettek (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15265376#comment-15265376
 ] 

Aljoscha Krettek commented on FLINK-3404:
-

I think implementing atomic checkpoint-and-stop is harder than it is made to 
look here. The problem is the synchronization of all parallel sources in 
stopping and checkpointing. I have some ideas on this but I think it is a 
bigger think that warrants at least a discussion on the dev list and possibly 
(probably) a design doc. (This is independent of the Kafka source.)

> Extend Kafka consumers with interface StoppableFunction
> ---
>
> Key: FLINK-3404
> URL: https://issues.apache.org/jira/browse/FLINK-3404
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Matthias J. Sax
>
> Kafka consumers are not stoppable right now. To make them stoppable, they 
> must implement {{StoppableFunction}}. Implementing method {{stop()}} must 
> ensure, that the consumer stops pulling new messages from Kafka and issues a 
> final checkpoint with the last offset. Afterwards, {{run()}} must return.
> When implementing this, keep in mind, that the gathered checkpoint might 
> later be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3404) Extend Kafka consumers with interface StoppableFunction

2016-04-29 Thread Matthias J. Sax (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15264146#comment-15264146
 ] 

Matthias J. Sax commented on FLINK-3404:


I think so. [~rmetzger] can you comment on this, as you are most familiar with 
KafkaSource.

> Extend Kafka consumers with interface StoppableFunction
> ---
>
> Key: FLINK-3404
> URL: https://issues.apache.org/jira/browse/FLINK-3404
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Matthias J. Sax
>
> Kafka consumers are not stoppable right now. To make them stoppable, they 
> must implement {{StoppableFunction}}. Implementing method {{stop()}} must 
> ensure, that the consumer stops pulling new messages from Kafka and issues a 
> final checkpoint with the last offset. Afterwards, {{run()}} must return.
> When implementing this, keep in mind, that the gathered checkpoint might 
> later be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3404) Extend Kafka consumers with interface StoppableFunction

2016-04-29 Thread Martin Liesenberg (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15263977#comment-15263977
 ] 

Martin Liesenberg commented on FLINK-3404:
--

I have been looking at this task recently and just want to make sure I 
understand it correctly: there are a {{close}} and a {{cancel}} method already. 
the difference of the {{stop}} method would be the issuing of the checkpoint, 
right?

> Extend Kafka consumers with interface StoppableFunction
> ---
>
> Key: FLINK-3404
> URL: https://issues.apache.org/jira/browse/FLINK-3404
> Project: Flink
>  Issue Type: Improvement
>  Components: Kafka Connector
>Reporter: Matthias J. Sax
>
> Kafka consumers are not stoppable right now. To make them stoppable, they 
> must implement {{StoppableFunction}}. Implementing method {{stop()}} must 
> ensure, that the consumer stops pulling new messages from Kafka and issues a 
> final checkpoint with the last offset. Afterwards, {{run()}} must return.
> When implementing this, keep in mind, that the gathered checkpoint might 
> later be used as a savepoint.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)