[jira] [Comment Edited] (FLINK-13306) flink-examples-streaming-gcp-pubsub is missing NOTICE

2019-07-26 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893794#comment-16893794
 ] 

Konstantin Knauf edited comment on FLINK-13306 at 7/26/19 12:30 PM:


This example is also not located in {{flink-examples-streaming}} + packing in 
{{flink-examples-build-helper}}, but instead all the code is located in 
{{flink-examples-build-helper}}.

I think, this should also be fixed, if we keep it.


was (Author: knaufk):
This example is also not located in \{flink-examples-streaming} + packing in 
\{flink-examples-build-helper}, but instead all the code is located in 
\{flink-examples-build-helper}.

I think, this should also be fixed, if we keep it.

> flink-examples-streaming-gcp-pubsub is missing NOTICE
> -
>
> Key: FLINK-13306
> URL: https://issues.apache.org/jira/browse/FLINK-13306
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Google Cloud PubSub, Examples
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
>
> The pubsub example is bundling various dependencies but is missing a NOTICE 
> file.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Comment Edited] (FLINK-13306) flink-examples-streaming-gcp-pubsub is missing NOTICE

2019-07-26 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893794#comment-16893794
 ] 

Konstantin Knauf edited comment on FLINK-13306 at 7/26/19 12:30 PM:


This example is also not located in \{flink-examples-streaming} + packing in 
\{flink-examples-build-helper}, but instead all the code is located in 
\{flink-examples-build-helper}.

I think, this should also be fixed, if we keep it.


was (Author: knaufk):
This example is also not located in `flink-examples-streaming` + packing in 
flink-examples-build-helper, but instead all the code is located in 
`flink-examples-build-helper`. 

I think, this should also be fixed, if we keep it.

> flink-examples-streaming-gcp-pubsub is missing NOTICE
> -
>
> Key: FLINK-13306
> URL: https://issues.apache.org/jira/browse/FLINK-13306
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Google Cloud PubSub, Examples
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
>
> The pubsub example is bundling various dependencies but is missing a NOTICE 
> file.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13306) flink-examples-streaming-gcp-pubsub is missing NOTICE

2019-07-26 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893794#comment-16893794
 ] 

Konstantin Knauf commented on FLINK-13306:
--

This example is also not located in `flink-examples-streaming` + packing in 
flink-examples-build-helper, but instead all the code is located in 
`flink-examples-build-helper`. 

I think, this should also be fixed, if we keep it.

> flink-examples-streaming-gcp-pubsub is missing NOTICE
> -
>
> Key: FLINK-13306
> URL: https://issues.apache.org/jira/browse/FLINK-13306
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Google Cloud PubSub, Examples
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Assignee: Chesnay Schepler
>Priority: Blocker
>
> The pubsub example is bundling various dependencies but is missing a NOTICE 
> file.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (FLINK-13417) Bump Zookeeper to 3.5.5

2019-07-25 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13417:
-
Affects Version/s: 1.9.0

> Bump Zookeeper to 3.5.5
> ---
>
> Key: FLINK-13417
> URL: https://issues.apache.org/jira/browse/FLINK-13417
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Coordination
>Affects Versions: 1.9.0
>Reporter: Konstantin Knauf
>Priority: Major
>
> User might want to secure their Zookeeper connection via SSL.
> This requires a Zookeeper version >= 3.5.1. We might as well try to bump it 
> to 3.5.5, which is the latest version. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (FLINK-13417) Bump Zookeeper to 3.5.5

2019-07-25 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13417:
-
Description: 
User might want to secure their Zookeeper connection via SSL.

This requires a Zookeeper version >= 3.5.1. We might as well try to bump it to 
3.5.5, which is the latest version. 

  was:
User might want to secure their Zookeeper connection via SSL.

This requires a Zookeeper version >= 3.5.1. We might as try to bump it to 
3.5.5. 


> Bump Zookeeper to 3.5.5
> ---
>
> Key: FLINK-13417
> URL: https://issues.apache.org/jira/browse/FLINK-13417
> Project: Flink
>  Issue Type: Improvement
>  Components: Runtime / Coordination
>Reporter: Konstantin Knauf
>Priority: Major
>
> User might want to secure their Zookeeper connection via SSL.
> This requires a Zookeeper version >= 3.5.1. We might as well try to bump it 
> to 3.5.5, which is the latest version. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Created] (FLINK-13417) Bump Zookeeper to 3.5.5

2019-07-25 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-13417:


 Summary: Bump Zookeeper to 3.5.5
 Key: FLINK-13417
 URL: https://issues.apache.org/jira/browse/FLINK-13417
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Konstantin Knauf


User might want to secure their Zookeeper connection via SSL.

This requires a Zookeeper version >= 3.5.1. We might as try to bump it to 
3.5.5. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (FLINK-13002) Expand Concept -> Glossary Section

2019-07-19 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13002:
-
Description: 
We use this ticket to collect terms, we would like to add to the Glossary in 
the future:
 * Snapshot
 * Checkpoint
 * Savepoint
 * Parallelism
 * Backpressure
 * TaskSlot

  was:
We use this ticket to collect terms, we would like to add to the Glossary in 
the future:
 * Snapshot
 * Checkpoint
 * Savepoint
 * Parallelism


> Expand Concept -> Glossary Section
> --
>
> Key: FLINK-13002
> URL: https://issues.apache.org/jira/browse/FLINK-13002
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> We use this ticket to collect terms, we would like to add to the Glossary in 
> the future:
>  * Snapshot
>  * Checkpoint
>  * Savepoint
>  * Parallelism
>  * Backpressure
>  * TaskSlot



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13002) Expand Concept -> Glossary Section

2019-07-19 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888773#comment-16888773
 ] 

Konstantin Knauf commented on FLINK-13002:
--

I can't. [~rmetzger] could you do this?

> Expand Concept -> Glossary Section
> --
>
> Key: FLINK-13002
> URL: https://issues.apache.org/jira/browse/FLINK-13002
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> We use this ticket to collect terms, we would like to add to the Glossary in 
> the future:
>  * Snapshot
>  * Checkpoint
>  * Savepoint
>  * Parallelism
>  * Backpressure
>  * TaskSlot



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (FLINK-13002) Expand Concept -> Glossary Section

2019-07-19 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13002:
-
Description: 
We use this ticket to collect terms, we would like to add to the Glossary in 
the future:
 * Snapshot
 * Checkpoint
 * Savepoint
 * Parallelism

  was:
We use this ticket to collect terms, we would like to add to the Glossary in 
the future: 

* Snapshot
* Checkpoint
* Savepoint


> Expand Concept -> Glossary Section
> --
>
> Key: FLINK-13002
> URL: https://issues.apache.org/jira/browse/FLINK-13002
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> We use this ticket to collect terms, we would like to add to the Glossary in 
> the future:
>  * Snapshot
>  * Checkpoint
>  * Savepoint
>  * Parallelism



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13037) Translate "Concepts -> Glossary" page into Chinese

2019-07-19 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888577#comment-16888577
 ] 

Konstantin Knauf commented on FLINK-13037:
--

[~highfei2...@126.com] Thank you!. I agree, the glossary is definitely not 
complete. We also have a separate Jira  [1] to collect missing terms. 

[1]  https://issues.apache.org/jira/browse/FLINK-13002

> Translate "Concepts -> Glossary" page into Chinese
> --
>
> Key: FLINK-13037
> URL: https://issues.apache.org/jira/browse/FLINK-13037
> Project: Flink
>  Issue Type: Sub-task
>  Components: chinese-translation, Documentation
>Reporter: Konstantin Knauf
>Assignee: Jeff Yang
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Translate Glossary page into Chinese: 
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
> The markdown file is located in {{docs/concepts/glossary.md}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-13037) Translate "Concepts -> Glossary" page into Chinese

2019-07-18 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887691#comment-16887691
 ] 

Konstantin Knauf commented on FLINK-13037:
--

[~highfei2...@126.com] I am not working on it. Feel free to assign yourself.

> Translate "Concepts -> Glossary" page into Chinese
> --
>
> Key: FLINK-13037
> URL: https://issues.apache.org/jira/browse/FLINK-13037
> Project: Flink
>  Issue Type: Sub-task
>  Components: chinese-translation, Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> Translate Glossary page into Chinese: 
> https://ci.apache.org/projects/flink/flink-docs-master/concepts/glossary.html
> The markdown file is located in {{docs/concepts/glossary.md}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-11654) Multiple transactional KafkaProducers writing to same cluster have clashing transaction IDs

2019-07-16 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886089#comment-16886089
 ] 

Konstantin Knauf commented on FLINK-11654:
--

[~becket_qin] The principle you described sounds good! I agree, that an 
exception might be the better way, but we would most likely break existing, 
working systems if we start failing fast here. Nevertheless, I am leaning 
towards exceptions.

We would have two tickets then, which can be implemented independently.: 

1. Expose {{transactionIdPrefix}} 
2. Fail Fast when overriding Kafka properties

What do you think? (cc [~aljoscha])


> Multiple transactional KafkaProducers writing to same cluster have clashing 
> transaction IDs
> ---
>
> Key: FLINK-11654
> URL: https://issues.apache.org/jira/browse/FLINK-11654
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kafka
>Affects Versions: 1.7.1
>Reporter: Jürgen Kreileder
>Priority: Major
> Fix For: 1.9.0
>
>
> We run multiple jobs on a cluster which write a lot to the same Kafka topic 
> from identically named sinks. When EXACTLY_ONCE semantic is enabled for the 
> KafkaProducers we run into a lot of ProducerFencedExceptions and all jobs go 
> into a restart cycle.
> Example exception from the Kafka log:
>  
> {code:java}
> [2019-02-18 18:05:28,485] ERROR [ReplicaManager broker=1] Error processing 
> append operation on partition finding-commands-dev-1-0 
> (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.ProducerFencedException: Producer's epoch is 
> no longer valid. There is probably another producer with a newer epoch. 483 
> (request epoch), 484 (server epoch)
> {code}
> The reason for this is the way FlinkKafkaProducer initializes the 
> TransactionalIdsGenerator:
> The IDs are only guaranteed to be unique for a single Job. But they can clash 
> between different Jobs (and Clusters).
>  
>  
> {code:java}
> --- 
> a/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java
> +++ 
> b/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java
> @@ -819,6 +819,7 @@ public class FlinkKafkaProducer
>                 nextTransactionalIdHintState = 
> context.getOperatorStateStore().getUnionListState(
>                         NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR);
>                 transactionalIdsGenerator = new TransactionalIdsGenerator(
> + // the prefix probably should include job id and maybe cluster id
>                         getRuntimeContext().getTaskName() + "-" + 
> ((StreamingRuntimeContext) getRuntimeContext()).getOperatorUniqueID(),
>                         getRuntimeContext().getIndexOfThisSubtask(),
>                         
> getRuntimeContext().getNumberOfParallelSubtasks(),{code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Comment Edited] (FLINK-12650) Redirect Users to Documentation Homepage if Requested Resource Does Not Exist

2019-07-16 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885927#comment-16885927
 ] 

Konstantin Knauf edited comment on FLINK-12650 at 7/16/19 8:16 AM:
---

[~sjwiesman] What is the status of this ticket? It this expected to work on the 
current master?


was (Author: knaufk):
[~sjwiesman] What is the status of this ticket? It this supposed to work on the 
current master?

> Redirect Users to Documentation Homepage if Requested Resource Does Not Exist
> -
>
> Key: FLINK-12650
> URL: https://issues.apache.org/jira/browse/FLINK-12650
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Seth Wiesman
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, if you go open a documentation page, which does not exist like 
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_timed.html
> the reply is a 404. In preparation of a larger restructuring, it would be 
> good if instead of a 404 we would redirect the user to the documentation 
> homepage for this version of the docs.
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_timed.html
>  -> https://ci.apache.org/projects/flink/flink-docs-release-1.8/



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-12650) Redirect Users to Documentation Homepage if Requested Resource Does Not Exist

2019-07-16 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885927#comment-16885927
 ] 

Konstantin Knauf commented on FLINK-12650:
--

[~sjwiesman] What is the status of this ticket? It this supposed to work on the 
current master?

> Redirect Users to Documentation Homepage if Requested Resource Does Not Exist
> -
>
> Key: FLINK-12650
> URL: https://issues.apache.org/jira/browse/FLINK-12650
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Seth Wiesman
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.9.0
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Currently, if you go open a documentation page, which does not exist like 
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_timed.html
> the reply is a 404. In preparation of a larger restructuring, it would be 
> good if instead of a 404 we would redirect the user to the documentation 
> homepage for this version of the docs.
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_timed.html
>  -> https://ci.apache.org/projects/flink/flink-docs-release-1.8/



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-11654) Multiple transactional KafkaProducers writing to same cluster have clashing transaction IDs

2019-07-15 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885020#comment-16885020
 ] 

Konstantin Knauf commented on FLINK-11654:
--

[~becket_qin] If we reuse the existing Kafka Producer configuration 
`transactional.id`, we would change its semantics. For us it is a prefix, but 
for Kafka it is the full id. So far, I think, we have not done something like 
this: either the configuration is simply passed on, or it is overwritten. 
Personally, I would rather add 
FlinkKafkaProducer#setTransactionalIdPrefix(String) then changing the meaning 
of the configuration of a dependency. Probably, we should even log a warning if 
the `transaction.id` is not null, and state that it is overwritten. The 
behavior would be as you described.

> Multiple transactional KafkaProducers writing to same cluster have clashing 
> transaction IDs
> ---
>
> Key: FLINK-11654
> URL: https://issues.apache.org/jira/browse/FLINK-11654
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kafka
>Affects Versions: 1.7.1
>Reporter: Jürgen Kreileder
>Priority: Major
> Fix For: 1.9.0
>
>
> We run multiple jobs on a cluster which write a lot to the same Kafka topic 
> from identically named sinks. When EXACTLY_ONCE semantic is enabled for the 
> KafkaProducers we run into a lot of ProducerFencedExceptions and all jobs go 
> into a restart cycle.
> Example exception from the Kafka log:
>  
> {code:java}
> [2019-02-18 18:05:28,485] ERROR [ReplicaManager broker=1] Error processing 
> append operation on partition finding-commands-dev-1-0 
> (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.ProducerFencedException: Producer's epoch is 
> no longer valid. There is probably another producer with a newer epoch. 483 
> (request epoch), 484 (server epoch)
> {code}
> The reason for this is the way FlinkKafkaProducer initializes the 
> TransactionalIdsGenerator:
> The IDs are only guaranteed to be unique for a single Job. But they can clash 
> between different Jobs (and Clusters).
>  
>  
> {code:java}
> --- 
> a/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java
> +++ 
> b/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java
> @@ -819,6 +819,7 @@ public class FlinkKafkaProducer
>                 nextTransactionalIdHintState = 
> context.getOperatorStateStore().getUnionListState(
>                         NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR);
>                 transactionalIdsGenerator = new TransactionalIdsGenerator(
> + // the prefix probably should include job id and maybe cluster id
>                         getRuntimeContext().getTaskName() + "-" + 
> ((StreamingRuntimeContext) getRuntimeContext()).getOperatorUniqueID(),
>                         getRuntimeContext().getIndexOfThisSubtask(),
>                         
> getRuntimeContext().getNumberOfParallelSubtasks(),{code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-11654) Multiple transactional KafkaProducers writing to same cluster have clashing transaction IDs

2019-07-12 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883786#comment-16883786
 ] 

Konstantin Knauf commented on FLINK-11654:
--

Sorry for joining the discussion late. It seems to me that users might 
generally want to specify the prefix for the transactional ids used. For 
example, only certain transactional ids might be authorized for a certain 
application (https://docs.confluent.io/current/kafka/authorization.html). I 
recently cam across a user, who had this requirement. Therefore I am leaning 
towards making {{transactionalIdPrefix}} an explicit parameter of the 
{{FlinkKafkaProducer}}.

> Multiple transactional KafkaProducers writing to same cluster have clashing 
> transaction IDs
> ---
>
> Key: FLINK-11654
> URL: https://issues.apache.org/jira/browse/FLINK-11654
> Project: Flink
>  Issue Type: Bug
>  Components: Connectors / Kafka
>Affects Versions: 1.7.1
>Reporter: Jürgen Kreileder
>Priority: Major
> Fix For: 1.9.0
>
>
> We run multiple jobs on a cluster which write a lot to the same Kafka topic 
> from identically named sinks. When EXACTLY_ONCE semantic is enabled for the 
> KafkaProducers we run into a lot of ProducerFencedExceptions and all jobs go 
> into a restart cycle.
> Example exception from the Kafka log:
>  
> {code:java}
> [2019-02-18 18:05:28,485] ERROR [ReplicaManager broker=1] Error processing 
> append operation on partition finding-commands-dev-1-0 
> (kafka.server.ReplicaManager)
> org.apache.kafka.common.errors.ProducerFencedException: Producer's epoch is 
> no longer valid. There is probably another producer with a newer epoch. 483 
> (request epoch), 484 (server epoch)
> {code}
> The reason for this is the way FlinkKafkaProducer initializes the 
> TransactionalIdsGenerator:
> The IDs are only guaranteed to be unique for a single Job. But they can clash 
> between different Jobs (and Clusters).
>  
>  
> {code:java}
> --- 
> a/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java
> +++ 
> b/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java
> @@ -819,6 +819,7 @@ public class FlinkKafkaProducer
>                 nextTransactionalIdHintState = 
> context.getOperatorStateStore().getUnionListState(
>                         NEXT_TRANSACTIONAL_ID_HINT_DESCRIPTOR);
>                 transactionalIdsGenerator = new TransactionalIdsGenerator(
> + // the prefix probably should include job id and maybe cluster id
>                         getRuntimeContext().getTaskName() + "-" + 
> ((StreamingRuntimeContext) getRuntimeContext()).getOperatorUniqueID(),
>                         getRuntimeContext().getIndexOfThisSubtask(),
>                         
> getRuntimeContext().getNumberOfParallelSubtasks(),{code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-11792) Make KafkaConsumer more resilient to Kafka Broker Failures

2019-07-12 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883611#comment-16883611
 ] 

Konstantin Knauf commented on FLINK-11792:
--

[~becket_qin] Thanks for the explanation. I think you are right. I also tried 
to reproduce the issue the user reported back then, but the consumer was always 
able to recover some time after the leadership change.

> Make KafkaConsumer more resilient to Kafka Broker Failures 
> ---
>
> Key: FLINK-11792
> URL: https://issues.apache.org/jira/browse/FLINK-11792
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kafka
>Affects Versions: 1.7.2
>Reporter: Konstantin Knauf
>Priority: Major
>
> When consuming from a topic with replication factor > 1, the 
> FlinkKafkaConsumer could continue reading from this topic, when a single 
> broker fails, by "simply" switching to the new leader `s for all lost 
> partitions after Kafka failover. Currently, the KafkaConsumer will most 
> likely throw in exception as topic metadata is only periodically fetched from 
> the Kafka cluster.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Resolved] (FLINK-11792) Make KafkaConsumer more resilient to Kafka Broker Failures

2019-07-12 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf resolved FLINK-11792.
--
Resolution: Not A Problem

> Make KafkaConsumer more resilient to Kafka Broker Failures 
> ---
>
> Key: FLINK-11792
> URL: https://issues.apache.org/jira/browse/FLINK-11792
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kafka
>Affects Versions: 1.7.2
>Reporter: Konstantin Knauf
>Priority: Major
>
> When consuming from a topic with replication factor > 1, the 
> FlinkKafkaConsumer could continue reading from this topic, when a single 
> broker fails, by "simply" switching to the new leader `s for all lost 
> partitions after Kafka failover. Currently, the KafkaConsumer will most 
> likely throw in exception as topic metadata is only periodically fetched from 
> the Kafka cluster.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (FLINK-11793) Make KafkaProducer more resilient to Kafka Broker Failures

2019-07-11 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-11793:
-
Description: Similar to FLINK-11792 we can make the FlinkKafkaProducer more 
resilient against Broker Failures by not immediately failing on certain 
{{KafkaException}}'s, but by retrying in case there is only a change in 
partition leadership.  (was: Similar to FLINK-11792 we can make the 
FlinkKafkaProducer more resilient against Broker Failures by not immediately 
failing on certain KafkaException's, but by retrying in case there is only a 
change in partition leadership.)

> Make KafkaProducer more resilient to Kafka Broker Failures
> --
>
> Key: FLINK-11793
> URL: https://issues.apache.org/jira/browse/FLINK-11793
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kafka
>Affects Versions: 1.7.2
>Reporter: Konstantin Knauf
>Priority: Major
>
> Similar to FLINK-11792 we can make the FlinkKafkaProducer more resilient 
> against Broker Failures by not immediately failing on certain 
> {{KafkaException}}'s, but by retrying in case there is only a change in 
> partition leadership.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Updated] (FLINK-11793) Make KafkaProducer more resilient to Kafka Broker Failures

2019-07-11 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-11793:
-
Description: Similar to FLINK-11792 we can make the FlinkKafkaProducer more 
resilient against Broker Failures by not immediately failing on certain 
KafkaException's, but by retrying in case there is only a change in partition 
leadership.  (was: Similar to FLINK-11792 we can make the FlinkKafkaProducer 
more resilient against Broker Failures by not immediately failing on certain 
KafkaException, but by retrying in case there is only a case in partition 
leadership.)

> Make KafkaProducer more resilient to Kafka Broker Failures
> --
>
> Key: FLINK-11793
> URL: https://issues.apache.org/jira/browse/FLINK-11793
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kafka
>Affects Versions: 1.7.2
>Reporter: Konstantin Knauf
>Priority: Major
>
> Similar to FLINK-11792 we can make the FlinkKafkaProducer more resilient 
> against Broker Failures by not immediately failing on certain 
> KafkaException's, but by retrying in case there is only a change in partition 
> leadership.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-11792) Make KafkaConsumer more resilient to Kafka Broker Failures

2019-07-11 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-11792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882771#comment-16882771
 ] 

Konstantin Knauf commented on FLINK-11792:
--

[~becket_qin] Thanks for having a look at this.  
https://kafka.apache.org/protocol.html#protocol_partitioning describes, that 
the client needs to manually check for metadata updates in case of a broker 
failure (all requests for a partition go to the leader). 
`KafkaConsumer:assign`, which we use, also states that " As such, there will be 
no rebalance operation triggered when group membership or cluster and topic 
metadata change." We update the metadata only in the 
`KafkaPartitionDiscoverer`, which is invoked periodically. I will try to find 
the actual stacktraces from back then. 

> Make KafkaConsumer more resilient to Kafka Broker Failures 
> ---
>
> Key: FLINK-11792
> URL: https://issues.apache.org/jira/browse/FLINK-11792
> Project: Flink
>  Issue Type: Improvement
>  Components: Connectors / Kafka
>Affects Versions: 1.7.2
>Reporter: Konstantin Knauf
>Priority: Major
>
> When consuming from a topic with replication factor > 1, the 
> FlinkKafkaConsumer could continue reading from this topic, when a single 
> broker fails, by "simply" switching to the new leader `s for all lost 
> partitions after Kafka failover. Currently, the KafkaConsumer will most 
> likely throw in exception as topic metadata is only periodically fetched from 
> the Kafka cluster.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)


[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground

2019-07-10 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882170#comment-16882170
 ] 

Konstantin Knauf commented on FLINK-12749:
--

In order to spin up the playground the users need a few files on their machine, 
namely the following:
{noformat}
playground/
├── docker-compose.yaml
└── conf
└── flink-conf.yaml
└── log4j-cli.properties
└── log4j-console.properties
{noformat}
The conf directory will eventually not be necessary anymore, while the 
docker-compose configuration is of course a prerequisite. I see two options to 
publish these configurations files:

*Option 1: Embedded in the Documentation*

The necessary files can be part of the documentation (expandable) and new users 
create the directory structure by copying the content from there.

Pro:
 * Correct Flink Version is automatically populated.
 * Playground is only maintained at one place, not additional dependencies 
(besides `library/flink` image and `StateMachineExample`)

Contra:
 * cumbersome and error-prone for the user

*Option 2: Dedicated Repository for Playground Setup*

We create a new Apache Flink repository `apache/flink-playgrounds` with the 
following directory structure:
{noformat}
flink-playground
└──flink-cluster-playground/
   ├── docker-compose.yaml
   └── conf
   └── flink-conf.yaml
   └── log4j-cli.properties
   └── log4j-console.properties
└──interactive-sql-playground/
{noformat}

The `interactive-sql-playground` will contain whatever is neccessry for 
FLINK-12750. 

We could have tags for each Flink version, so that we can automatically link to 
the correct tag in the `flink-playgrounds` repository from the respective 
documentation version.

Pro:
 * user only needs to checkout this small repository and is ready to go.
 * we can make sure the playground setup always works, because we explicitly 
bump the Flink version once the new images are ready and we have tested the 
playground

Contra:
 * additional repository to maintain next to the documentation

Any opinions? [~fhueske] [~rmetzger] [~sjwiesman]

> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based 
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
> including a step-by-step guide for some common commands (job submission, 
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be 
> the most maintainable, I think. It would be preferable, if we don't need to 
> host any custom images for this, but can rely on the existing plain Flink 
> images.
> * Which Flink jobs to use? An updated version 
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} 
> might be a good option as it can with or without Kafka and contains a data 
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash 
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ 
> seems to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Resolved] (FLINK-12748) Getting Started - Flink Overview

2019-07-10 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf resolved FLINK-12748.
--
Resolution: Won't Do

> Getting Started - Flink Overview
> 
>
> Key: FLINK-12748
> URL: https://issues.apache.org/jira/browse/FLINK-12748
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is 
> *  Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we should add a 1-2 page introduction & overview of Apache 
> Flink including among other things an overview of the API Stack (DataStream 
> API & SQL/Table API), an introduction to *stateful* stream processing and 
> Flink's role in an overall stream processing architecture.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12748) Getting Started - Flink Overview

2019-07-10 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881885#comment-16881885
 ] 

Konstantin Knauf commented on FLINK-12748:
--

I talked to [~sjwiesman] about this issue and we think that we don't need this 
section in the documenation as we have this on the project page already. We 
will link to the project page from the index page of the documenation though. 
Closing this for now.

> Getting Started - Flink Overview
> 
>
> Key: FLINK-12748
> URL: https://issues.apache.org/jira/browse/FLINK-12748
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is 
> *  Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we should add a 1-2 page introduction & overview of Apache 
> Flink including among other things an overview of the API Stack (DataStream 
> API & SQL/Table API), an introduction to *stateful* stream processing and 
> Flink's role in an overall stream processing architecture.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-09 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.
{noformat}
2019-07-05 13:16:49,809 INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Custom Source (1/1) (1a22bba845872431e8695fc8f3793fcd) switched from 
RUNNING to FAILED.
java.lang.Exception: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:194)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:121)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:268)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:376)
taskmanager_1 | at org.apache.flink.runtime.taskmanager.Task.run(Task.java:675)
taskmanager_1 | at java.lang.Thread.run(Thread.java:748)
taskmanager_1 | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1 | at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1 | at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1 | at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:817)
taskmanager_1 | at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:144)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:150)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:781)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:656)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
taskmanager_1 | at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1120)
taskmanager_1 | at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1 | at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1 | at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1 | ... 1 more
{noformat}
Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the stacktrace looks 
like this:
{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
{noformat}
Rebased on {{86bee8679112e76372a84083b1af18722644e1a0}} without 
{{ExceptionUtils.rethrowException(error, 

[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-09 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.
{noformat}
2019-07-05 13:16:49,809 INFO org.apache.flink.runtime.taskmanager.Task - 
Source: Custom Source (1/1) (1a22bba845872431e8695fc8f3793fcd) switched from 
RUNNING to FAILED.
java.lang.Exception: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:194)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:121)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:268)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:376)
taskmanager_1 | at org.apache.flink.runtime.taskmanager.Task.run(Task.java:675)
taskmanager_1 | at java.lang.Thread.run(Thread.java:748)
taskmanager_1 | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1 | at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1 | at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1 | at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:817)
taskmanager_1 | at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:144)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:150)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:781)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:656)
taskmanager_1 | at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
taskmanager_1 | at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1120)
taskmanager_1 | at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1 | at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1 | at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1 | ... 1 more
{noformat}
Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the stacktrace looks 
like this:
{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
{noformat}
Rebased on a {{86bee8679112e76372a84083b1af18722644e1a0}} without 
{{ExceptionUtils.rethrowException(error, 

[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-09 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.

{noformat}
taskmanager_1  | 2019-07-09 09:00:35,498 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (8568229c7efcddf75545a503bdb737f8) switched from RUNNING to FAILED.
taskmanager_1  | org.apache.flink.util.FlinkException: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.pollNext(Handover.java:85)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:131)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:711)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:95)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:59)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:102)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:724)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
taskmanager_1  | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:184)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |... 1 more
{noformat}

Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the stacktrace looks 
like this:

{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 

[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-09 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.

{noformat}
taskmanager_1  | 2019-07-09 09:00:35,498 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (8568229c7efcddf75545a503bdb737f8) switched from RUNNING to FAILED.
taskmanager_1  | org.apache.flink.util.FlinkException: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.pollNext(Handover.java:85)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.runFetchLoop(KafkaFetcher.java:131)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:711)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:95)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:59)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:102)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:335)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:724)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
taskmanager_1  | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:184)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |... 1 more


Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the stacktrace looks 
like this:

{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 

[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-09 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.

{noformat}
taskmanager_1  | 2019-07-05 14:46:51,348 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (6dc79bc2d9654d140a2bab51456658de) switched from RUNNING to FAILED.
taskmanager_1  | java.lang.Exception: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:194)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:121)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:685)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:515)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
taskmanager_1  | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:818)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:134)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:144)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:150)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:808)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:683)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1130)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |... 1 more
{noformat}

Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the stacktrace looks 
like this:

{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 

[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-09 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.

{noformat}
taskmanager_1  | 2019-07-05 14:46:51,348 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (6dc79bc2d9654d140a2bab51456658de) switched from RUNNING to FAILED.
taskmanager_1  | java.lang.Exception: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:194)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:121)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:685)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:515)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
taskmanager_1  | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:818)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:134)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:144)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:150)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:808)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:683)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1130)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |... 1 more
{noformat}

Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the stacktrace looks 
like this:

{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 

[jira] [Updated] (FLINK-13123) Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation

2019-07-09 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13123:
-
Description: 
Currently, the REST API and CLI around stopping and cancelling jobs are not 
aligned in terms of terminology and the differences between {{cancel}} and 
{{job}} are not as clear as they could be.

I would like to make the following changes to the CLI: 

* add deprecation warning for {{cancel -s}} command and redirect users to  
{{stop}}
* rename {{-s}} of {{stop}} command to {{-p}} for savepoint location. Emphasize 
that this is optional, as a savepoint is taken in any case

I would like to make the following changes to the REST API: 

* Rename {{stop-with-savepoint}} to {{stop}} 
* Rename "endOfEventTime" to "drain" in accordance with the CLI

  was:
Currently, the REST API and CLI around stopping and cancelling jobs are not 
aligned in terms of terminology and the differences between {{cancel}} and 
{{job}} are not as clear as they could be.

I would like to make the following changes to the CLI: 

* add deprecation warning for {{cancel -s}} command and redirect users to  
{{stop}}
* rename {{-s}} of {{stop}} command to {{-p}} for savepoint location. Emphasize 
that this is optional, as a savepoint is taken in any case

I would like to make the following changes to the REST API: 

* Rename {{stop-with-savepoint}} to {{stop}} 
* Log a Deprecation warning with /savepoints is used in conjuction with cancel. 
* Rename "endOfEventTime" to "drain" in accordance with the CLI


> Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation
> --
>
> Key: FLINK-13123
> URL: https://issues.apache.org/jira/browse/FLINK-13123
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> Currently, the REST API and CLI around stopping and cancelling jobs are not 
> aligned in terms of terminology and the differences between {{cancel}} and 
> {{job}} are not as clear as they could be.
> I would like to make the following changes to the CLI: 
> * add deprecation warning for {{cancel -s}} command and redirect users to  
> {{stop}}
> * rename {{-s}} of {{stop}} command to {{-p}} for savepoint location. 
> Emphasize that this is optional, as a savepoint is taken in any case
> I would like to make the following changes to the REST API: 
> * Rename {{stop-with-savepoint}} to {{stop}} 
> * Rename "endOfEventTime" to "drain" in accordance with the CLI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-13123) Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation

2019-07-08 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880385#comment-16880385
 ] 

Konstantin Knauf commented on FLINK-13123:
--

The "-s" command for "cancel" has two effects: 
* a savepoint is created
* and optionally changing the savepoint path

The current "-s" command for "stop" is only used to optionally change the 
savepoint path. I think "-p" makes it clearer that this is different to "-s" 
for cancel and it is only used to specify the savepoint path.

> Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation
> --
>
> Key: FLINK-13123
> URL: https://issues.apache.org/jira/browse/FLINK-13123
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> Currently, the REST API and CLI around stopping and cancelling jobs are not 
> aligned in terms of terminology and the differences between {{cancel}} and 
> {{job}} are not as clear as they could be.
> I would like to make the following changes to the CLI: 
> * add deprecation warning for {{cancel -s}} command and redirect users to  
> {{stop}}
> * rename {{-s}} of {{stop}} command to {{-p}} for savepoint location. 
> Emphasize that this is optional, as a savepoint is taken in any case
> I would like to make the following changes to the REST API: 
> * Rename {{stop-with-savepoint}} to {{stop}} 
> * Log a Deprecation warning with /savepoints is used in conjuction with 
> cancel. 
> * Rename "endOfEventTime" to "drain" in accordance with the CLI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-13151) Rename JobManager (Process) to Flink Master

2019-07-08 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-13151:


 Summary: Rename JobManager (Process) to Flink Master
 Key: FLINK-13151
 URL: https://issues.apache.org/jira/browse/FLINK-13151
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, Runtime / Coordination
Reporter: Konstantin Knauf


In FLINK-12652 we coined the term "Flink Master" to describe the master process 
of a Flink cluster (containing JobManagers, Dispatcher and ResourceManager). 
This needs to be reflected in the rest of the documentation, scripts (where 
possible) and code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-13151) Rename "JobManager (Process)" to "Flink Master"

2019-07-08 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13151:
-
Summary: Rename "JobManager (Process)" to "Flink Master"  (was: Rename 
JobManager (Process) to Flink Master)

> Rename "JobManager (Process)" to "Flink Master"
> ---
>
> Key: FLINK-13151
> URL: https://issues.apache.org/jira/browse/FLINK-13151
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation, Runtime / Coordination
>Reporter: Konstantin Knauf
>Priority: Major
>
> In FLINK-12652 we coined the term "Flink Master" to describe the master 
> process of a Flink cluster (containing JobManagers, Dispatcher and 
> ResourceManager). This needs to be reflected in the rest of the 
> documentation, scripts (where possible) and code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-08 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.

{noformat}
taskmanager_1  | 2019-07-05 14:46:51,348 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (6dc79bc2d9654d140a2bab51456658de) switched from RUNNING to FAILED.
taskmanager_1  | java.lang.Exception: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:194)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:121)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:685)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:515)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
taskmanager_1  | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:818)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:134)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:144)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:150)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:808)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:683)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1130)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |... 1 more
{noformat}

Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the stacktrace looks 
like this:

{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 

[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-08 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13124:
-
Description: 
When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.

{noformat}
taskmanager_1  | 2019-07-05 14:46:51,348 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (6dc79bc2d9654d140a2bab51456658de) switched from RUNNING to FAILED.
taskmanager_1  | java.lang.Exception: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:194)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:121)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:685)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:515)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
taskmanager_1  | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:818)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:134)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:144)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:150)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:808)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:683)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1130)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |... 1 more
{noformat}

Rebased on a `86bee8679112e76372a84083b1af18722644e1a0` the StackTrace looks 
like this:

{noformat}
taskmanager_1  | 2019-07-08 13:01:01,287 INFO  
org.apache.flink.runtime.taskmanager.Task - Source: Custom 
Source (1/1) (54504e053415cee82a2a7b4293d325e9) switched from RUNNING to FAILED.
taskmanager_1  | 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:814)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:108)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:114)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:729)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:604)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:124)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1151)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 

[jira] [Updated] (FLINK-13123) Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation

2019-07-08 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13123:
-
Description: 
Currently, the REST API and CLI around stopping and cancelling jobs are not 
aligned in terms of terminology and the differences between {{cancel}} and 
{{job}} are not as clear as they could be.

I would like to make the following changes to the CLI: 

* add deprecation warning for {{cancel -s}} command and redirect users to  
{{stop}}
* rename {{-s}} of {{stop}} command to {{-p}} for savepoint location. Emphasize 
that this is optional, as a savepoint is taken in any case

I would like to make the following changes to the REST API: 

* Rename {{stop-with-savepoint}} to {{stop}} 
* Log a Deprecation warning with /savepoints is used in conjuction with cancel. 
* Rename "endOfEventTime" to "drain" in accordance with the CLI

  was:
Currently, the REST API and CLI around stopping and cancelling jobs are not 
aligned in terms of terminology and the differences between {{cancel}} and 
{{job}} are not as clear as they could be.

I would like to make the following changes to the CLI: 

* add deprecation warning for {{cancel -s}} command and redirect users to  
{{stop}}
* rename {{-s}} of {{stop}} command to {{-l}} for savepoint location. Emphasize 
that this is optional, as a savepoint is taken in anycase

I would like to make the following changes to the REST API: 

* Rename {{stop-with-savepoint}} to {{stop}} 
* Log a Deprecation warning with /savepoints is used in conjuction with cancel. 
* Rename "endOfEventTime" to "drain" in accordance with the CLI


> Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation
> --
>
> Key: FLINK-13123
> URL: https://issues.apache.org/jira/browse/FLINK-13123
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> Currently, the REST API and CLI around stopping and cancelling jobs are not 
> aligned in terms of terminology and the differences between {{cancel}} and 
> {{job}} are not as clear as they could be.
> I would like to make the following changes to the CLI: 
> * add deprecation warning for {{cancel -s}} command and redirect users to  
> {{stop}}
> * rename {{-s}} of {{stop}} command to {{-p}} for savepoint location. 
> Emphasize that this is optional, as a savepoint is taken in any case
> I would like to make the following changes to the REST API: 
> * Rename {{stop-with-savepoint}} to {{stop}} 
> * Log a Deprecation warning with /savepoints is used in conjuction with 
> cancel. 
> * Rename "endOfEventTime" to "drain" in accordance with the CLI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-05 Thread Konstantin Knauf (JIRA)
eamTask.finishTask(SourceStreamTask.java:150)
> taskmanager_1  |  at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:808)
> taskmanager_1  |  at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:683)
> taskmanager_1  |  at 
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
> taskmanager_1  |  at 
> org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1130)
> taskmanager_1  |  at 
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> taskmanager_1  |  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> taskmanager_1  |  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> taskmanager_1  |  ... 1 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-13124) Stop fails with Universal Kafka Consumer

2019-07-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-13124:


 Summary: Stop fails with Universal Kafka Consumer
 Key: FLINK-13124
 URL: https://issues.apache.org/jira/browse/FLINK-13124
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.9.0
Reporter: Konstantin Knauf


When running the {{StateMachineExample}} (with the universal Kafka connector 
instead of 0.10) the Job always crashes with the following exception, when 
stopping it with {{flink stop }}.

{noformat}
2019-07-05 13:16:49,809 INFO  org.apache.flink.runtime.taskmanager.Task 
- Source: Custom Source (1/1) (1a22bba845872431e8695fc8f3793fcd) 
switched from RUNNING to FAILED.
java.lang.Exception: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.checkThrowSourceExecutionException(SourceStreamTask.java:194)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.performDefaultAction(SourceStreamTask.java:121)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:268)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:376)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task.run(Task.java:675)
taskmanager_1  |at java.lang.Thread.run(Thread.java:748)
taskmanager_1  | Caused by: 
org.apache.flink.streaming.connectors.kafka.internal.Handover$ClosedException
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.Handover.close(Handover.java:182)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.internal.KafkaFetcher.cancel(KafkaFetcher.java:175)
taskmanager_1  |at 
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.cancel(FlinkKafkaConsumerBase.java:817)
taskmanager_1  |at 
org.apache.flink.streaming.api.operators.StreamSource.cancel(StreamSource.java:124)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.cancelTask(SourceStreamTask.java:144)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.finishTask(SourceStreamTask.java:150)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:781)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpoint(StreamTask.java:656)
taskmanager_1  |at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.triggerCheckpoint(SourceStreamTask.java:160)
taskmanager_1  |at 
org.apache.flink.runtime.taskmanager.Task$1.run(Task.java:1120)
taskmanager_1  |at 
java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
taskmanager_1  |at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
taskmanager_1  |... 1 more
{noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (FLINK-13123) Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation

2019-07-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf reassigned FLINK-13123:


Assignee: Konstantin Knauf

> Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation
> --
>
> Key: FLINK-13123
> URL: https://issues.apache.org/jira/browse/FLINK-13123
> Project: Flink
>  Issue Type: Improvement
>Affects Versions: 1.9.0
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> Currently, the REST API and CLI around stopping and cancelling jobs are not 
> aligned in terms of terminology and the differences between {{cancel}} and 
> {{job}} are not as clear as they could be.
> I would like to make the following changes to the CLI: 
> * add deprecation warning for {{cancel -s}} command and redirect users to  
> {{stop}}
> * rename {{-s}} of {{stop}} command to {{-l}} for savepoint location. 
> Emphasize that this is optional, as a savepoint is taken in anycase
> I would like to make the following changes to the REST API: 
> * Rename {{stop-with-savepoint}} to {{stop}} 
> * Log a Deprecation warning with /savepoints is used in conjuction with 
> cancel. 
> * Rename "endOfEventTime" to "drain" in accordance with the CLI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-13123) Align Stop/Cancel Commands in CLI and REST Interface and Improve Documentation

2019-07-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-13123:


 Summary: Align Stop/Cancel Commands in CLI and REST Interface and 
Improve Documentation
 Key: FLINK-13123
 URL: https://issues.apache.org/jira/browse/FLINK-13123
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.9.0
Reporter: Konstantin Knauf


Currently, the REST API and CLI around stopping and cancelling jobs are not 
aligned in terms of terminology and the differences between {{cancel}} and 
{{job}} are not as clear as they could be.

I would like to make the following changes to the CLI: 

* add deprecation warning for {{cancel -s}} command and redirect users to  
{{stop}}
* rename {{-s}} of {{stop}} command to {{-l}} for savepoint location. Emphasize 
that this is optional, as a savepoint is taken in anycase

I would like to make the following changes to the REST API: 

* Rename {{stop-with-savepoint}} to {{stop}} 
* Log a Deprecation warning with /savepoints is used in conjuction with cancel. 
* Rename "endOfEventTime" to "drain" in accordance with the CLI



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12746) Getting Started - Project Setup and DataStream Example Walkthrough

2019-07-02 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12746:
-
Description: 
The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
** Java
** Scala
** Python
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we should add "Project Setup" and "Quickstarts -> Example 
Walkthrough - DataStream API", which covers everything what we have today. This 
will replace the current "Tutorials" and "Examples" section, which can be 
removed as part of this ticket as well.

  was:
The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we should add "Project Setup" and "Quickstarts -> Example 
Walkthrough - DataStream API", which covers everything what we have today. This 
will replace the current "Tutorials" and "Examples" section, which can be 
removed as part of this ticket as well.


> Getting Started - Project Setup and DataStream Example Walkthrough
> --
>
> Key: FLINK-12746
> URL: https://issues.apache.org/jira/browse/FLINK-12746
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Seth Wiesman
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The planned structure for the new Getting Started Guide is 
> *  Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we should add "Project Setup" and "Quickstarts -> Example 
> Walkthrough - DataStream API", which covers everything what we have today. 
> This will replace the current "Tutorials" and "Examples" section, which can 
> be removed as part of this ticket as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12748) Getting Started - Flink Overview

2019-07-02 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12748:
-
Description: 
The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
** Java
** Scala
** Python
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we should add a 1-2 page introduction & overview of Apache Flink 
including among other things an overview of the API Stack (DataStream API & 
SQL/Table API), an introduction to *stateful* stream processing and Flink's 
role in an overall stream processing architecture.

  was:
The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we should add a 1-2 page introduction & overview of Apache Flink 
including among other things an overview of the API Stack (DataStream API & 
SQL/Table API), an introduction to *stateful* stream processing and Flink's 
role in an overall stream processing architecture.


> Getting Started - Flink Overview
> 
>
> Key: FLINK-12748
> URL: https://issues.apache.org/jira/browse/FLINK-12748
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is 
> *  Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we should add a 1-2 page introduction & overview of Apache 
> Flink including among other things an overview of the API Stack (DataStream 
> API & SQL/Table API), an introduction to *stateful* stream processing and 
> Flink's role in an overall stream processing architecture.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12750) Gettting Started - Docker Playground - Interactive SQL Playground

2019-07-02 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12750:
-
Description: 
The planned structure for the new Getting Started Guide is

* Flink Overview (~ two pages)
* Project Setup
** Java
** Scala
** Python
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster) and 
an SQL-Client. 

The general setup should be in line with FLINK-12749. 

**Open Questions**
* Where to host the SQL Client image? Can we somehow also use existing plain 
Flink images?

  was:
The planned structure for the new Getting Started Guide is

* Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster) and 
an SQL-Client. 

The general setup should be in line with FLINK-12749. 

**Open Questions**
* Where to host the SQL Client image? Can we somehow also use existing plain 
Flink images?


> Gettting Started - Docker Playground - Interactive SQL Playground
> -
>
> Key: FLINK-12750
> URL: https://issues.apache.org/jira/browse/FLINK-12750
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based 
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster) and 
> an SQL-Client. 
> The general setup should be in line with FLINK-12749. 
> **Open Questions**
> * Where to host the SQL Client image? Can we somehow also use existing plain 
> Flink images?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12747) Getting Started - Table API Example Walkthrough

2019-07-02 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12747:
-
Description: 
The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
** Java
** Scala
** Python
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

This tickets adds the Example Walkthrough for the Table API, which should 
follow the same structure as the DataStream Example (FLINK-12746), which needs 
to be completed first.

  was:
The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

This tickets adds the Example Walkthrough for the Table API, which should 
follow the same structure as the DataStream Example (FLINK-12746), which needs 
to be completed first.


> Getting Started - Table API Example Walkthrough
> ---
>
> Key: FLINK-12747
> URL: https://issues.apache.org/jira/browse/FLINK-12747
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Seth Wiesman
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The planned structure for the new Getting Started Guide is 
> *  Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> This tickets adds the Example Walkthrough for the Table API, which should 
> follow the same structure as the DataStream Example (FLINK-12746), which 
> needs to be completed first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground

2019-07-02 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12749:
-
Description: 
The planned structure for the new Getting Started Guide is

* Flink Overview (~ two pages)
* Project Setup
** Java
** Scala
** Python
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
including a step-by-step guide for some common commands (job submission, 
savepoints, etc).

*Some Open Questions:*
* Which Flink images to use? `library/flink` with dynamic properties would be 
the most maintainable, I think. It would be preferable, if we don't need to 
host any custom images for this, but can rely on the existing plain Flink 
images.
* Which Flink jobs to use? An updated version 
{{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might 
be a good option as it can with or without Kafka and contains a data generator 
writing to Kafka already (see next questions).
* How to get data into Kafka? Maybe just provide a small bash script/one-liner 
to produce into Kafka topic or see question above.
* Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems 
to be well-maintained and is openly available.

  was:
The planned structure for the new Getting Started Guide is

* Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
including a step-by-step guide for some common commands (job submission, 
savepoints, etc).

*Some Open Questions:*
* Which Flink images to use? `library/flink` with dynamic properties would be 
the most maintainable, I think. It would be preferable, if we don't need to 
host any custom images for this, but can rely on the existing plain Flink 
images.
* Which Flink jobs to use? An updated version 
{{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might 
be a good option as it can with or without Kafka and contains a data generator 
writing to Kafka already (see next questions).
* How to get data into Kafka? Maybe just provide a small bash script/one-liner 
to produce into Kafka topic or see question above.
* Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems 
to be well-maintained and is openly available.


> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based 
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
> including a step-by-step guide for some common commands (job submission, 
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be 
> the most maintainable, I think. It would be preferable, if we don't need to 
> host any custom images for this, but can rely on the existing plain Flink 
> images.
> * Which Flink jobs to use? An updated version 
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} 
> might be a good option as it can with or without Kafka and contains a data 
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash 
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ 
> seems to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-13037) Translate "Concepts -> Glossary" page into Chinese

2019-07-02 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876798#comment-16876798
 ] 

Konstantin Knauf commented on FLINK-13037:
--

[~jark] Sure, done!

> Translate "Concepts -> Glossary" page into Chinese
> --
>
> Key: FLINK-13037
> URL: https://issues.apache.org/jira/browse/FLINK-13037
> Project: Flink
>  Issue Type: Sub-task
>  Components: chinese-translation, Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-13037) Translate "Concepts -> Glossary" page into Chinese

2019-07-01 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-13037:


 Summary: Translate "Concepts -> Glossary" page into Chinese
 Key: FLINK-13037
 URL: https://issues.apache.org/jira/browse/FLINK-13037
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-7129) Support dynamically changing CEP patterns

2019-06-26 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873490#comment-16873490
 ] 

Konstantin Knauf commented on FLINK-7129:
-

[~Mostafa01] Generally speaking, you can implement a dynamic pattern matching 
yourself with a `KeyedBroadProcessFunction`, where patterns are broadcasted and 
the `KeyedStream` is checked against these patterns. The complexity of this 
approach depends mostly on how complex/flexible the patterns/rule are you would 
like to support.

> Support dynamically changing CEP patterns
> -
>
> Key: FLINK-7129
> URL: https://issues.apache.org/jira/browse/FLINK-7129
> Project: Flink
>  Issue Type: New Feature
>  Components: Library / CEP
>Reporter: Dawid Wysakowicz
>Priority: Major
>
> An umbrella task for introducing mechanism for injecting patterns through 
> coStream



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (FLINK-7129) Support dynamically changing CEP patterns

2019-06-26 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873490#comment-16873490
 ] 

Konstantin Knauf edited comment on FLINK-7129 at 6/26/19 4:24 PM:
--

[~Mostafa01] Generally speaking, you can implement a dynamic pattern matching 
yourself with a {{KeyedBroadcastProcessFunction}}, where patterns are 
broadcasted and the {{KeyedStream}} is checked against these patterns. The 
complexity of this approach depends mostly on how complex/flexible the 
patterns/rule are you would like to support.


was (Author: knaufk):
[~Mostafa01] Generally speaking, you can implement a dynamic pattern matching 
yourself with a `KeyedBroadProcessFunction`, where patterns are broadcasted and 
the `KeyedStream` is checked against these patterns. The complexity of this 
approach depends mostly on how complex/flexible the patterns/rule are you would 
like to support.

> Support dynamically changing CEP patterns
> -
>
> Key: FLINK-7129
> URL: https://issues.apache.org/jira/browse/FLINK-7129
> Project: Flink
>  Issue Type: New Feature
>  Components: Library / CEP
>Reporter: Dawid Wysakowicz
>Priority: Major
>
> An umbrella task for introducing mechanism for injecting patterns through 
> coStream



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-13002) Expand Concept -> Glossary Section

2019-06-26 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873436#comment-16873436
 ] 

Konstantin Knauf commented on FLINK-13002:
--

I think, the glossary should include terms which are often confused, not 
well-defined or generally used in a lot of different places. 

> Expand Concept -> Glossary Section
> --
>
> Key: FLINK-13002
> URL: https://issues.apache.org/jira/browse/FLINK-13002
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> We use this ticket to collect terms, we would like to add to the Glossary in 
> the future: 
> * Snapshot
> * Checkpoint
> * Savepoint



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12921) Flink Job Scheduling

2019-06-26 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873370#comment-16873370
 ] 

Konstantin Knauf commented on FLINK-12921:
--

[~vim] Thanks for opening this ticket. The documentation is not correct here. 
The behavior changed after FLIP-6.

I have linked a related ticket to enable the old behavior again.

> Flink Job Scheduling
> 
>
> Key: FLINK-12921
> URL: https://issues.apache.org/jira/browse/FLINK-12921
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataSet, API / DataStream
>Affects Versions: 1.8.0
> Environment: flink 1.8.0
> jdk8
>Reporter: vim
>Priority: Major
>
> This is an example from flink-docs:
> Consider a program with a data source, a _MapFunction_, and a 
> _ReduceFunction_. The source and MapFunction are executed with a parallelism 
> of 4, while the ReduceFunction is executed with a parallelism of 3. A 
> pipeline consists of the sequence Source - Map - Reduce. On a cluster with 2 
> TaskManagers with 3 slots each, the program will be executed as described 
> below.
> !https://ci.apache.org/projects/flink/flink-docs-release-1.8/fig/slots.svg!
> But after I tried, I found that it was not like this. My result is 
> TaskManager 1 used 3 slot, but TaskManager 2 just used 1 slot. Who else has 
> tested this example?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12752) Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint

2019-06-26 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12752:
-
Description: 
In case of a standalone job cluster setup, we would like to generate random a 
{{JobID}} for every job, but the {{JobID}} nevertheless needs to stay constant 
between JobManager process restarts. 

For this, I would like to add an additional command line options for the 
{{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. {{job-id-seed}} and 
{{job-id}} are mutually exclusive.

* --jobId
* --jobId-seed
* (default)

On Kubernetes, this new command line argument would need to be set to a 
property which is stable over Pod Restarts but changes for different K8s Jobs 
or K8s Deployments.

*Open Questions*
* what available information to use a seed for deployments on Kubernetes?


  was:
In case of a standalone job cluster setup, we would like to generate random a 
{{JobID}} for every job, but the {{JobID}} nevertheless needs to stay constant 
between JobManager process restarts. 

For this, I would like to add an additional command line options for the 
{{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually specified 
jobId would still take precedence (not breaking current behavior): 

* --jobId
* --jobId-seed
* (default)

On Kubernetes, this new command line argument would need to be set to a 
property which is stable over Pod Restarts but changes for different K8s Jobs 
or K8s Deployments.

*Open Questions*
* what available information to use a seed for deployments on Kubernetes?



> Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint
> -
>
> Key: FLINK-12752
> URL: https://issues.apache.org/jira/browse/FLINK-12752
> Project: Flink
>  Issue Type: Sub-task
>  Components: Deployment / Docker
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In case of a standalone job cluster setup, we would like to generate random a 
> {{JobID}} for every job, but the {{JobID}} nevertheless needs to stay 
> constant between JobManager process restarts. 
> For this, I would like to add an additional command line options for the 
> {{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. {{job-id-seed}} and 
> {{job-id}} are mutually exclusive.
> * --jobId
> * --jobId-seed
> * (default)
> On Kubernetes, this new command line argument would need to be set to a 
> property which is stable over Pod Restarts but changes for different K8s Jobs 
> or K8s Deployments.
> *Open Questions*
> * what available information to use a seed for deployments on Kubernetes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-13002) Expand Concept -> Glossary Section

2019-06-26 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-13002:
-
Component/s: Documentation

> Expand Concept -> Glossary Section
> --
>
> Key: FLINK-13002
> URL: https://issues.apache.org/jira/browse/FLINK-13002
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> We use this ticket to collect terms, we would like to add to the Glossary in 
> the future: 
> * Snapshot
> * Checkpoint
> * Savepoint



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-13002) Expand Concept -> Glossary Section

2019-06-26 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-13002:


 Summary: Expand Concept -> Glossary Section
 Key: FLINK-13002
 URL: https://issues.apache.org/jira/browse/FLINK-13002
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


We use this ticket to collect terms, we would like to add to the Glossary in 
the future: 

* Snapshot
* Checkpoint
* Savepoint



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12980) Getting Started - Add Top-Level Section to Existing Documentation

2019-06-25 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12980:


 Summary: Getting Started - Add Top-Level Section to Existing 
Documentation
 Key: FLINK-12980
 URL: https://issues.apache.org/jira/browse/FLINK-12980
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf
Assignee: Konstantin Knauf






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12752) Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint

2019-06-24 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16870878#comment-16870878
 ] 

Konstantin Knauf commented on FLINK-12752:
--

[~1u0] Yes, originally I hoped there is some field in Kubernetes, which could 
be used as the seed. After looking into it more, I now think, that no such 
field exists. Nevertheless, I think, this is valuable because of the free-form 
format and the provided templates. Additionally, this makes injecting the 
{{JobID}} from outside more flexible for other environment, too.

> Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint
> -
>
> Key: FLINK-12752
> URL: https://issues.apache.org/jira/browse/FLINK-12752
> Project: Flink
>  Issue Type: Sub-task
>  Components: Deployment / Docker
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> In case of a standalone job cluster setup, we would like to generate random a 
> {{JobID}} for every job, but the {{JobID}} nevertheless needs to stay 
> constant between JobManager process restarts. 
> For this, I would like to add an additional command line options for the 
> {{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually 
> specified jobId would still take precedence (not breaking current behavior): 
> * --jobId
> * --jobId-seed
> * (default)
> On Kubernetes, this new command line argument would need to be set to a 
> property which is stable over Pod Restarts but changes for different K8s Jobs 
> or K8s Deployments.
> *Open Questions*
> * what available information to use a seed for deployments on Kubernetes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12885) Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation

2019-06-19 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867673#comment-16867673
 ] 

Konstantin Knauf commented on FLINK-12885:
--

I have a PR ready for this based on the pending PR for FLINK-12752. Once this 
is merged I will open a PR for this ticket.

> Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation
> --
>
> Key: FLINK-12885
> URL: https://issues.apache.org/jira/browse/FLINK-12885
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12885) Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation

2019-06-19 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12885:
-
Component/s: Documentation

> Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation
> --
>
> Key: FLINK-12885
> URL: https://issues.apache.org/jira/browse/FLINK-12885
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (FLINK-12885) Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation

2019-06-18 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf reassigned FLINK-12885:


Assignee: Konstantin Knauf

> Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation
> --
>
> Key: FLINK-12885
> URL: https://issues.apache.org/jira/browse/FLINK-12885
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12885) Rename "Job Cluster" to "Application Cluster" in READMEs and Documentation

2019-06-18 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12885:


 Summary: Rename "Job Cluster" to "Application Cluster" in READMEs 
and Documentation
 Key: FLINK-12885
 URL: https://issues.apache.org/jira/browse/FLINK-12885
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-11787) Add web ui metrics reporting workaround into Flink Kubernetes resource definitions (Flink 1.7)

2019-06-13 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863109#comment-16863109
 ] 

Konstantin Knauf commented on FLINK-11787:
--

[~Zentol] Thanks. Makes sense.

> Add web ui metrics reporting workaround into Flink Kubernetes resource 
> definitions (Flink 1.7)
> --
>
> Key: FLINK-11787
> URL: https://issues.apache.org/jira/browse/FLINK-11787
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 1.7.2
>Reporter: Alex
>Assignee: Alex
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.7.3
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> It may take some effort to implement a proper solution for FLINK-11127 (that 
> includes high availability case).
> As a quick fix (workaround) for FLINK-11127 in case of Kubernetes 
> environments, it may be helpful to update the documentation and example 
> Kubernetes resource definitions to have the workaround present.
> *Note:* this is only for Flink version {{1.7}}. For later Flink versions, a 
> new {{taskmanager.network.bind-policy}} configuration option already "embeds" 
> this workaround.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-11787) Add web ui metrics reporting workaround into Flink Kubernetes resource definitions (Flink 1.7)

2019-06-13 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-11787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16863079#comment-16863079
 ] 

Konstantin Knauf commented on FLINK-11787:
--

[~till.rohrmann] why has this not been merged to `master` as well, too? It 
looks like https://issues.apache.org/jira/browse/FLINK-11127 will still take a 
while and in the meantime this is the proper workaround, isn't it?

> Add web ui metrics reporting workaround into Flink Kubernetes resource 
> definitions (Flink 1.7)
> --
>
> Key: FLINK-11787
> URL: https://issues.apache.org/jira/browse/FLINK-11787
> Project: Flink
>  Issue Type: Sub-task
>Affects Versions: 1.7.2
>Reporter: Alex
>Assignee: Alex
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.7.3
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> It may take some effort to implement a proper solution for FLINK-11127 (that 
> includes high availability case).
> As a quick fix (workaround) for FLINK-11127 in case of Kubernetes 
> environments, it may be helpful to update the documentation and example 
> Kubernetes resource definitions to have the workaround present.
> *Note:* this is only for Flink version {{1.7}}. For later Flink versions, a 
> new {{taskmanager.network.bind-policy}} configuration option already "embeds" 
> this workaround.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12752) Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint

2019-06-06 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857851#comment-16857851
 ] 

Konstantin Knauf commented on FLINK-12752:
--

I had a quick chat with [~plucas] offline and it seems there is not really 
anything in Kubernetes, which we could use. So, the plan would be for the time 
being to specify the {{jobId-seed}} in the pod spec of the K8s Deployment/Job 
explicitly. 

> Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint
> -
>
> Key: FLINK-12752
> URL: https://issues.apache.org/jira/browse/FLINK-12752
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> In case of a standalone job cluster setup, we would like to generate random a 
> {{JobID}} for every job, but the {{JobID}} nevertheless needs to stay 
> constant between JobManager process restarts. 
> For this, I would like to add an additional command line options for the 
> {{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually 
> specified jobId would still take precedence (not breaking current behavior): 
> * --jobId
> * --jobId-seed
> * (default)
> On Kubernetes, this new command line argument would need to be set to a 
> property which is stable over Pod Restarts but changes for different K8s Jobs 
> or K8s Deployments.
> *Open Questions*
> * what available information to use a seed for deployments on Kubernetes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12752) Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint

2019-06-06 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12752:
-
Component/s: Deployment / Docker

> Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint
> -
>
> Key: FLINK-12752
> URL: https://issues.apache.org/jira/browse/FLINK-12752
> Project: Flink
>  Issue Type: Sub-task
>  Components: Deployment / Docker
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> In case of a standalone job cluster setup, we would like to generate random a 
> {{JobID}} for every job, but the {{JobID}} nevertheless needs to stay 
> constant between JobManager process restarts. 
> For this, I would like to add an additional command line options for the 
> {{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually 
> specified jobId would still take precedence (not breaking current behavior): 
> * --jobId
> * --jobId-seed
> * (default)
> On Kubernetes, this new command line argument would need to be set to a 
> property which is stable over Pod Restarts but changes for different K8s Jobs 
> or K8s Deployments.
> *Open Questions*
> * what available information to use a seed for deployments on Kubernetes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12752) Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint

2019-06-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12752:
-
Description: 
In case of a standalone job cluster setup, we would like to generate random a 
{{JobID}} for every job, but the {{JobID}} nevertheless needs to stay constant 
between JobManager process restarts. 

For this, I would like to add an additional command line options for the 
{{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually specified 
jobId would still take precedence (not breaking current behavior): 

* --jobId
* --jobId-seed
* (default)

On Kubernetes, this new command line argument would need to be set to a 
property which is stable over Pod Restarts but changes for different K8s Jobs 
or K8s Deployments.

*Open Questions*
* what available information to use a seed for deployments on Kubernetes?


  was:
In case of a standalone job cluster setup, we would like to generate random a 
{{JobID}} for every job, but the {{JobID}} nevertheless needs to stay constant 
between JobManager process restarts. 

For this, I would like to add an additional command line options for the 
{{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually specified 
jobId would still take precedence (not breaking current behavior): 

* --jobId
* --jobId-seed
* (default)

On Kubernetes, this new command line argument would need to be set to a 
property which is stable over Pod Restarts but changes for different K8s Jobs 
or K8s Deployments.




> Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint
> -
>
> Key: FLINK-12752
> URL: https://issues.apache.org/jira/browse/FLINK-12752
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> In case of a standalone job cluster setup, we would like to generate random a 
> {{JobID}} for every job, but the {{JobID}} nevertheless needs to stay 
> constant between JobManager process restarts. 
> For this, I would like to add an additional command line options for the 
> {{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually 
> specified jobId would still take precedence (not breaking current behavior): 
> * --jobId
> * --jobId-seed
> * (default)
> On Kubernetes, this new command line argument would need to be set to a 
> property which is stable over Pod Restarts but changes for different K8s Jobs 
> or K8s Deployments.
> *Open Questions*
> * what available information to use a seed for deployments on Kubernetes?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12752) Add Option to Pass Seed for JobID Hash for StandaloneJobClusterEntrypoint

2019-06-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12752:


 Summary: Add Option to Pass Seed for JobID Hash for 
StandaloneJobClusterEntrypoint
 Key: FLINK-12752
 URL: https://issues.apache.org/jira/browse/FLINK-12752
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf
Assignee: Konstantin Knauf


In case of a standalone job cluster setup, we would like to generate random a 
{{JobID}} for every job, but the {{JobID}} nevertheless needs to stay constant 
between JobManager process restarts. 

For this, I would like to add an additional command line options for the 
{{StandaloneJobClusterEntrypoint}} called {{jobId-seed}}. A manually specified 
jobId would still take precedence (not breaking current behavior): 

* --jobId
* --jobId-seed
* (default)

On Kubernetes, this new command line argument would need to be set to a 
property which is stable over Pod Restarts but changes for different K8s Jobs 
or K8s Deployments.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground

2019-06-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf reassigned FLINK-12749:


Assignee: Konstantin Knauf

> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based 
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
> including a step-by-step guide for some common commands (job submission, 
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be 
> the most maintainable, I think. It would be preferable, if we don't need to 
> host any custom images for this, but can rely on the existing plain Flink 
> images.
> * Which Flink jobs to use? An updated version 
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} 
> might be a good option as it can with or without Kafka and contains a data 
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash 
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ 
> seems to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12750) Gettting Started - Docker Playground - Interactive SQL Playground

2019-06-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12750:


 Summary: Gettting Started - Docker Playground - Interactive SQL 
Playground
 Key: FLINK-12750
 URL: https://issues.apache.org/jira/browse/FLINK-12750
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


The planned structure for the new Getting Started Guide is

* Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster) and 
an SQL-Client. 

The general setup should be in line with FLINK-12749. 

**Open Questions**
* Where to host the SQL Client image? Can we somehow also use existing plain 
Flink images?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground

2019-06-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12749:
-
Description: 
The planned structure for the new Getting Started Guide is

*Flink Overview (~ two pages)
*Project Setup
*Quickstarts
**Example Walkthrough - Table API / SQL
**Example Walkthrough - DataStream API
*Docker Playgrounds
**Flink Cluster Playground
**Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
including a step-by-step guide for some common commands (job submission, 
savepoints, etc).

*Some Open Questions:*
* Which Flink images to use? `library/flink` with dynamic properties would be 
the most maintainable, I think. It would be preferable, if we don't need to 
host any custom images for this, but can rely on the existing plain Flink 
images.
* Which Flink jobs to use? An updated version 
{{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might 
be a good option as it can with or without Kafka and contains a data generator 
writing to Kafka already (see next questions).
* How to get data into Kafka? Maybe just provide a small bash script/one-liner 
to produce into Kafka topic or see question above.
* Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems 
to be well-maintained and is openly available.

  was:
The planned structure for the new Getting Started Guide is

Flink Overview (~ two pages)
Project Setup
Quickstarts
Example Walkthrough - Table API / SQL
Example Walkthrough - DataStream API
Docker Playgrounds
Flink Cluster Playground
Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
including a step-by-step guide for some common commands (job submission, 
savepoints, etc).

*Some Open Questions:*
* Which Flink images to use? `library/flink` with dynamic properties would be 
the most maintainable, I think. It would be preferable, if we don't need to 
host any custom images for this, but can rely on the existing plain Flink 
images.
* Which Flink jobs to use? An updated version 
{{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might 
be a good option as it can with or without Kafka and contains a data generator 
writing to Kafka already (see next questions).
* How to get data into Kafka? Maybe just provide a small bash script/one-liner 
to produce into Kafka topic or see question above.
* Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems 
to be well-maintained and is openly available.


> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> *Flink Overview (~ two pages)
> *Project Setup
> *Quickstarts
> **Example Walkthrough - Table API / SQL
> **Example Walkthrough - DataStream API
> *Docker Playgrounds
> **Flink Cluster Playground
> **Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based 
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
> including a step-by-step guide for some common commands (job submission, 
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be 
> the most maintainable, I think. It would be preferable, if we don't need to 
> host any custom images for this, but can rely on the existing plain Flink 
> images.
> * Which Flink jobs to use? An updated version 
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} 
> might be a good option as it can with or without Kafka and contains a data 
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash 
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ 
> seems to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground

2019-06-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12749:
-
Description: 
The planned structure for the new Getting Started Guide is

* Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
including a step-by-step guide for some common commands (job submission, 
savepoints, etc).

*Some Open Questions:*
* Which Flink images to use? `library/flink` with dynamic properties would be 
the most maintainable, I think. It would be preferable, if we don't need to 
host any custom images for this, but can rely on the existing plain Flink 
images.
* Which Flink jobs to use? An updated version 
{{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might 
be a good option as it can with or without Kafka and contains a data generator 
writing to Kafka already (see next questions).
* How to get data into Kafka? Maybe just provide a small bash script/one-liner 
to produce into Kafka topic or see question above.
* Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems 
to be well-maintained and is openly available.

  was:
The planned structure for the new Getting Started Guide is

*Flink Overview (~ two pages)
*Project Setup
*Quickstarts
**Example Walkthrough - Table API / SQL
**Example Walkthrough - DataStream API
*Docker Playgrounds
**Flink Cluster Playground
**Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
including a step-by-step guide for some common commands (job submission, 
savepoints, etc).

*Some Open Questions:*
* Which Flink images to use? `library/flink` with dynamic properties would be 
the most maintainable, I think. It would be preferable, if we don't need to 
host any custom images for this, but can rely on the existing plain Flink 
images.
* Which Flink jobs to use? An updated version 
{{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might 
be a good option as it can with or without Kafka and contains a data generator 
writing to Kafka already (see next questions).
* How to get data into Kafka? Maybe just provide a small bash script/one-liner 
to produce into Kafka topic or see question above.
* Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems 
to be well-maintained and is openly available.


> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based 
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
> including a step-by-step guide for some common commands (job submission, 
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be 
> the most maintainable, I think. It would be preferable, if we don't need to 
> host any custom images for this, but can rely on the existing plain Flink 
> images.
> * Which Flink jobs to use? An updated version 
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} 
> might be a good option as it can with or without Kafka and contains a data 
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash 
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ 
> seems to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground

2019-06-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12749:
-
Summary: Getting Started - Docker Playgrounds - Flink Cluster Playground  
(was: FLINK-12748 - Docker Playgrounds - Flink Cluster Playground)

> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
>  Issue Type: Sub-task
>Reporter: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> Flink Overview (~ two pages)
> Project Setup
> Quickstarts
> Example Walkthrough - Table API / SQL
> Example Walkthrough - DataStream API
> Docker Playgrounds
> Flink Cluster Playground
> Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based 
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
> including a step-by-step guide for some common commands (job submission, 
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be 
> the most maintainable, I think. It would be preferable, if we don't need to 
> host any custom images for this, but can rely on the existing plain Flink 
> images.
> * Which Flink jobs to use? An updated version 
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} 
> might be a good option as it can with or without Kafka and contains a data 
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash 
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ 
> seems to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12749) FLINK-12748 - Docker Playgrounds - Flink Cluster Playground

2019-06-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12749:


 Summary: FLINK-12748 - Docker Playgrounds - Flink Cluster 
Playground
 Key: FLINK-12749
 URL: https://issues.apache.org/jira/browse/FLINK-12749
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


The planned structure for the new Getting Started Guide is

Flink Overview (~ two pages)
Project Setup
Quickstarts
Example Walkthrough - Table API / SQL
Example Walkthrough - DataStream API
Docker Playgrounds
Flink Cluster Playground
Flink Interactive SQL Playground

In this ticket we add the Flink Cluster Playground, a docker-compose based 
setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster), 
including a step-by-step guide for some common commands (job submission, 
savepoints, etc).

*Some Open Questions:*
* Which Flink images to use? `library/flink` with dynamic properties would be 
the most maintainable, I think. It would be preferable, if we don't need to 
host any custom images for this, but can rely on the existing plain Flink 
images.
* Which Flink jobs to use? An updated version 
{{org.apache.flink.streaming.examples.statemachine.StateMachineExample}} might 
be a good option as it can with or without Kafka and contains a data generator 
writing to Kafka already (see next questions).
* How to get data into Kafka? Maybe just provide a small bash script/one-liner 
to produce into Kafka topic or see question above.
* Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/ seems 
to be well-maintained and is openly available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12748) Getting Started - Flink Overview

2019-06-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12748:


 Summary: Getting Started - Flink Overview
 Key: FLINK-12748
 URL: https://issues.apache.org/jira/browse/FLINK-12748
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we should add a 1-2 page introduction & overview of Apache Flink 
including among other things an overview of the API Stack (DataStream API & 
SQL/Table API), an introduction to *stateful* stream processing and Flink's 
role in an overall stream processing architecture.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12747) Getting Started - Table API Example Walkthrough

2019-06-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12747:


 Summary: Getting Started - Table API Example Walkthrough
 Key: FLINK-12747
 URL: https://issues.apache.org/jira/browse/FLINK-12747
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

This tickets adds the Example Walkthrough for the Table API, which should 
follow the same structure as the DataStream Example (FLINK-12746), which needs 
to be completed first.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12746) Getting Started - Project Setup and DataStream Example Walkthrough

2019-06-05 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12746:


 Summary: Getting Started - Project Setup and DataStream Example 
Walkthrough
 Key: FLINK-12746
 URL: https://issues.apache.org/jira/browse/FLINK-12746
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


The planned structure for the new Getting Started Guide is 

*  Flink Overview (~ two pages)
* Project Setup
* Quickstarts
** Example Walkthrough - Table API / SQL
** Example Walkthrough - DataStream API
* Docker Playgrounds
** Flink Cluster Playground
** Flink Interactive SQL Playground

In this ticket we should add "Project Setup" and "Quickstarts -> Example 
Walkthrough - DataStream API", which covers everything what we have today. This 
will replace the current "Tutorials" and "Examples" section, which can be 
removed as part of this ticket as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12652) Glossary - Add Glossary to Concepts Section of Documentation

2019-06-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12652:
-
Summary: Glossary - Add Glossary to Concepts Section of Documentation  
(was: Add Glossary to Concepts Section of Documentation)

> Glossary - Add Glossary to Concepts Section of Documentation
> 
>
> Key: FLINK-12652
> URL: https://issues.apache.org/jira/browse/FLINK-12652
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12724) Glossary - Add Links to new Concepts Section to Glossary

2019-06-05 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12724:
-
Summary: Glossary - Add Links to new Concepts Section to Glossary  (was: 
Add Links to new Concepts Section to Glossary)

> Glossary - Add Links to new Concepts Section to Glossary
> 
>
> Key: FLINK-12724
> URL: https://issues.apache.org/jira/browse/FLINK-12724
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> Once we have reworked the Concepts section, we should add references to to 
> the glossary. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12724) Add Links to new Concepts Section to Glossary

2019-06-04 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12724:


 Summary: Add Links to new Concepts Section to Glossary
 Key: FLINK-12724
 URL: https://issues.apache.org/jira/browse/FLINK-12724
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Konstantin Knauf


Once we have reworked the Concepts section, we should add references to to the 
glossary. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12691) Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime

2019-05-31 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12691:
-
Description: 
A user that I have recently been working with has the requirement to change the 
capacity and possibly the timeout during the runtime of a job. 

-Conceptually, this should be possibly by just passing these parameters to the 
{{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
only apply to future records and the change in the queue capacity would take 
effect immediately.-
 



  was:
A user that I have recently been working with has the requirement to change the 
capacity and possibly the timeout during the runtime of a job. 

-Conceptually, this should be possibly by just passing these parameters to the 
{{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
only apply to future records and the change in the queue capacity would take 
effect immediately.
-
 




> Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime
> --
>
> Key: FLINK-12691
> URL: https://issues.apache.org/jira/browse/FLINK-12691
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Reporter: Konstantin Knauf
>Priority: Minor
>
> A user that I have recently been working with has the requirement to change 
> the capacity and possibly the timeout during the runtime of a job. 
> -Conceptually, this should be possibly by just passing these parameters to 
> the {{RichAsyncFunction}} via its RuntimeContext. The change of the timeout 
> would only apply to future records and the change in the queue capacity would 
> take effect immediately.-
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12691) Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime

2019-05-31 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12691:
-
Description: 
A user that I have recently been working with has the requirement to change the 
capacity and possibly the timeout during the runtime of a job. 

-Conceptually, this should be possibly by just passing these parameters to the 
{{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
only apply to future records and the change in the queue capacity would take 
effect immediately.
-
 



  was:
A user that I have recently been working with has the requirement to change the 
capacity and possibly the timeout during the runtime of a job. 

~Conceptually, this should be possibly by just passing these parameters to the 
{{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
only apply to future records and the change in the queue capacity would take 
effect immediately.~

 




> Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime
> --
>
> Key: FLINK-12691
> URL: https://issues.apache.org/jira/browse/FLINK-12691
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Reporter: Konstantin Knauf
>Priority: Minor
>
> A user that I have recently been working with has the requirement to change 
> the capacity and possibly the timeout during the runtime of a job. 
> -Conceptually, this should be possibly by just passing these parameters to 
> the {{RichAsyncFunction}} via its RuntimeContext. The change of the timeout 
> would only apply to future records and the change in the queue capacity would 
> take effect immediately.
> -
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12691) Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime

2019-05-31 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12691:
-
Description: 
A user that I have recently been working with has the requirement to change the 
capacity and possibly the timeout during the runtime of a job. 

~Conceptually, this should be possibly by just passing these parameters to the 
{{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
only apply to future records and the change in the queue capacity would take 
effect immediately.~

 



  was:
A user that I have recently been working with has the requirement to change the 
capacity and possibly the timeout during the runtime of a job. 

Conceptually, this should be possibly by just passing these parameters to the 
{{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
only apply to future records and the change in the queue capacity would take 
effect immediately.

 




> Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime
> --
>
> Key: FLINK-12691
> URL: https://issues.apache.org/jira/browse/FLINK-12691
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Reporter: Konstantin Knauf
>Priority: Minor
>
> A user that I have recently been working with has the requirement to change 
> the capacity and possibly the timeout during the runtime of a job. 
> ~Conceptually, this should be possibly by just passing these parameters to 
> the {{RichAsyncFunction}} via its RuntimeContext. The change of the timeout 
> would only apply to future records and the change in the queue capacity would 
> take effect immediately.~
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12691) Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime

2019-05-31 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853028#comment-16853028
 ] 

Konstantin Knauf commented on FLINK-12691:
--

[~yanghua] I thought about this again and a) it is not as easy as I thought and 
b) from an API perspective the {{RuntimeContext}} would probably also not be 
nice, because it generally does not contain operator specific methods. 
Optimally, we would have a dedicated {{Context}}, which was passed to 
{{asyncInvoke}}, but this would break the API. Any ideas? 



> Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime
> --
>
> Key: FLINK-12691
> URL: https://issues.apache.org/jira/browse/FLINK-12691
> Project: Flink
>  Issue Type: Improvement
>  Components: API / DataStream
>Reporter: Konstantin Knauf
>Priority: Minor
>
> A user that I have recently been working with has the requirement to change 
> the capacity and possibly the timeout during the runtime of a job. 
> Conceptually, this should be possibly by just passing these parameters to the 
> {{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
> only apply to future records and the change in the queue capacity would take 
> effect immediately.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12691) Make Queue Capacity and Timeout of AsyncWaitOperator changeable during runtime

2019-05-31 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12691:


 Summary: Make Queue Capacity and Timeout of AsyncWaitOperator 
changeable during runtime
 Key: FLINK-12691
 URL: https://issues.apache.org/jira/browse/FLINK-12691
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Konstantin Knauf


A user that I have recently been working with has the requirement to change the 
capacity and possibly the timeout during the runtime of a job. 

Conceptually, this should be possibly by just passing these parameters to the 
{{RichAsyncFunction}} via its RuntimeContext. The change of the timeout would 
only apply to future records and the change in the queue capacity would take 
effect immediately.

 





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (FLINK-12546) Base Docker images on `library/flink`

2019-05-29 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf reassigned FLINK-12546:


Assignee: Konstantin Knauf

> Base Docker images on `library/flink`
> -
>
> Key: FLINK-12546
> URL: https://issues.apache.org/jira/browse/FLINK-12546
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Docker
>Affects Versions: 1.8.0
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> Currently, Flink users turn to two different places when looking for 
> "official" community Docker images. 
> * https://github.com/docker-flink/docker-flink (Flink community maintained, 
> no official Apache releases) or https://hub.docker.com/_/flink/ 
> * The tooling and Dockerfile in the {{flink-container}} component
> While users should turn to the Flink images on Docker Hub in general, the 
> {{flink-container}} component is used by many users, because it contains some 
> tooling to build images for the {{StandaloneJobClusterEntrypoint}}. Overall, 
> this causes confusion for users and the community needs to maintain two 
> Dockerfiles, which build Flink images FROM alpine or debian.
> Therefore, I propose to change the tooling ({{build.sh}}) in 
> {{flink-container}} to have only two options: 
> a) {{from-release}}, which uses `library/flink` as base image and basically 
> only adds the user jar.  (<--- for Flink users)
> b)  {{from-local-dist}}, which also uses `library/flink` as base image but 
> replaces the flink-dist by the local flink-dist ( <--- for Flink developer)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12546) Base Docker images on `library/flink`

2019-05-29 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850643#comment-16850643
 ] 

Konstantin Knauf commented on FLINK-12546:
--

Hi [~till.rohrmann], Hi [~yunta]

I think, we should merge (job and session clusters) this in {{flink-container}} 
eventually and drop {{flink-contrib/docker-flink}}. In the scope of this ticket 
I agree with [~till.rohrmann] to simply base both on the same images 
{{libary/flink}}. 

Cheers, 

Konstantin

> Base Docker images on `library/flink`
> -
>
> Key: FLINK-12546
> URL: https://issues.apache.org/jira/browse/FLINK-12546
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Docker
>Affects Versions: 1.8.0
>Reporter: Konstantin Knauf
>Priority: Major
>
> Currently, Flink users turn to two different places when looking for 
> "official" community Docker images. 
> * https://github.com/docker-flink/docker-flink (Flink community maintained, 
> no official Apache releases) or https://hub.docker.com/_/flink/ 
> * The tooling and Dockerfile in the {{flink-container}} component
> While users should turn to the Flink images on Docker Hub in general, the 
> {{flink-container}} component is used by many users, because it contains some 
> tooling to build images for the {{StandaloneJobClusterEntrypoint}}. Overall, 
> this causes confusion for users and the community needs to maintain two 
> Dockerfiles, which build Flink images FROM alpine or debian.
> Therefore, I propose to change the tooling ({{build.sh}}) in 
> {{flink-container}} to have only two options: 
> a) {{from-release}}, which uses `library/flink` as base image and basically 
> only adds the user jar.  (<--- for Flink users)
> b)  {{from-local-dist}}, which also uses `library/flink` as base image but 
> replaces the flink-dist by the local flink-dist ( <--- for Flink developer)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Assigned] (FLINK-12652) Add Glossary to Concepts Section of Documentation

2019-05-28 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf reassigned FLINK-12652:


Assignee: Konstantin Knauf

> Add Glossary to Concepts Section of Documentation
> -
>
> Key: FLINK-12652
> URL: https://issues.apache.org/jira/browse/FLINK-12652
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Updated] (FLINK-12651) Add "Style Guide" to Documentation Contribution Guide

2019-05-28 Thread Konstantin Knauf (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantin Knauf updated FLINK-12651:
-
Component/s: Documentation

> Add "Style Guide" to Documentation Contribution Guide
> -
>
> Key: FLINK-12651
> URL: https://issues.apache.org/jira/browse/FLINK-12651
> Project: Flink
>  Issue Type: Sub-task
>  Components: Documentation
>Reporter: Konstantin Knauf
>Priority: Major
>
> As part of this FLIP we would like to align on a few standards & guidelines 
> for our documentation.
> The current guide only focuses on the process and technical aspects:
> https://flink.apache.org/contribute-documentation.html
> https://flink.apache.org/how-to-contribute.html#contribute-documentation
> We would like to expand on this for things like:
> * when to use "note" or "attention" tags
> * where the table of contents should be placed on a page
> * when to split up pages
> * how to address the user
> * when to add code samples in Java & Scala
> * ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12652) Add Glossary to Concepts Section of Documentation

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12652:


 Summary: Add Glossary to Concepts Section of Documentation
 Key: FLINK-12652
 URL: https://issues.apache.org/jira/browse/FLINK-12652
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Konstantin Knauf






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12651) Add "Style Guide" to Documentation Contribution Guide

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12651:


 Summary: Add "Style Guide" to Documentation Contribution Guide
 Key: FLINK-12651
 URL: https://issues.apache.org/jira/browse/FLINK-12651
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


As part of this FLIP we would like to align on a few standards & guidelines for 
our documentation.

The current guide only focuses on the process and technical aspects:

https://flink.apache.org/contribute-documentation.html
https://flink.apache.org/how-to-contribute.html#contribute-documentation

We would like to expand on this for things like:
* when to use "note" or "attention" tags
* where the table of contents should be placed on a page
* when to split up pages
* how to address the user
* when to add code samples in Java & Scala
* ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12650) Redirect Users to Documentation Homepage if Requested Resource Does Not Exist

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12650:


 Summary: Redirect Users to Documentation Homepage if Requested 
Resource Does Not Exist
 Key: FLINK-12650
 URL: https://issues.apache.org/jira/browse/FLINK-12650
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf


Currently, if you go open a documentation page, which does not exist like 

https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_timed.html

the reply is a 404. In preparation of a larger restructuring, it would be good 
if instead of a 404 we would redirect the user to the documentation homepage 
for this version of the docs.


https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/event_timed.html
 -> https://ci.apache.org/projects/flink/flink-docs-release-1.8/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12639) FLIP-42: Rework Documentation

2019-05-28 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12639:


 Summary: FLIP-42: Rework Documentation
 Key: FLINK-12639
 URL: https://issues.apache.org/jira/browse/FLINK-12639
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Konstantin Knauf
Assignee: Konstantin Knauf






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12546) Base Docker images on `library/flink`

2019-05-27 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848742#comment-16848742
 ] 

Konstantin Knauf commented on FLINK-12546:
--

[~yunta] When basing the images on {{library/flink}} docker will do the caching 
for us automatically :)

https://hub.docker.com/_/flink/ also has images for older Flink versions. Still 
I wonder, how far back we need to add support in {{build.sh}} if it makes 
things more complicated. In my opinion, tooling to build the current and the 
last version of Flink (1.8/1.7) should suffice. If a user needs to build older 
Flink images, she can always check out an older Flink release for the older 
scripts.

[~till.rohrmann][~yunta] What do you think?



> Base Docker images on `library/flink`
> -
>
> Key: FLINK-12546
> URL: https://issues.apache.org/jira/browse/FLINK-12546
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Docker
>Affects Versions: 1.8.0
>Reporter: Konstantin Knauf
>Priority: Major
>
> Currently, Flink users turn to two different places when looking for 
> "official" community Docker images. 
> * https://github.com/docker-flink/docker-flink (Flink community maintained, 
> no official Apache releases) or https://hub.docker.com/_/flink/ 
> * The tooling and Dockerfile in the {{flink-container}} component
> While users should turn to the Flink images on Docker Hub in general, the 
> {{flink-container}} component is used by many users, because it contains some 
> tooling to build images for the {{StandaloneJobClusterEntrypoint}}. Overall, 
> this causes confusion for users and the community needs to maintain two 
> Dockerfiles, which build Flink images FROM alpine or debian.
> Therefore, I propose to change the tooling ({{build.sh}}) in 
> {{flink-container}} to have only two options: 
> a) {{from-release}}, which uses `library/flink` as base image and basically 
> only adds the user jar.  (<--- for Flink users)
> b)  {{from-local-dist}}, which also uses `library/flink` as base image but 
> replaces the flink-dist by the local flink-dist ( <--- for Flink developer)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12381) W/o HA, upon a full restart, checkpointing crashes

2019-05-27 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848736#comment-16848736
 ] 

Konstantin Knauf commented on FLINK-12381:
--

[~till.rohrmann] I opened  a PR for the first step for a subtask issue 
https://issues.apache.org/jira/browse/FLINK-12617.

> W/o HA, upon a full restart, checkpointing crashes
> --
>
> Key: FLINK-12381
> URL: https://issues.apache.org/jira/browse/FLINK-12381
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.8.0
> Environment: Same as FLINK-\{12379, 12377, 12376}
>Reporter: Henrik
>Assignee: Konstantin Knauf
>Priority: Major
>
> {code:java}
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> 'gs://example_bucket/flink/checkpoints//chk-16/_metadata'
>  already exists
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.createChannel(GoogleHadoopOutputStream.java:85)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.(GoogleHadoopOutputStream.java:74)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:797)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:807)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.(FsCheckpointMetadataOutputStream.java:65)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointStorageLocation.createMetadataOutputStream(FsCheckpointStorageLocation.java:104)
>     at 
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:259)
>     at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:829)
>     ... 8 more
> {code}
> Instead, it should either just overwrite the checkpoint or fail to start the 
> job completely. Partial and undefined failure is not what should happen.
>  
> Repro:
>  # Set up a single purpose job cluster (which could use much better docs btw!)
>  # Let it run with GCS checkpointing for a while with rocksdb/gs://example
>  # Kill it
>  # Start it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12617) StandaloneJobClusterEntrypoint should default to random JobID for non-HA setups

2019-05-24 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12617:


 Summary: StandaloneJobClusterEntrypoint should default to random 
JobID for non-HA setups 
 Key: FLINK-12617
 URL: https://issues.apache.org/jira/browse/FLINK-12617
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Configuration
Affects Versions: 1.8.0
Reporter: Konstantin Knauf
Assignee: Konstantin Knauf






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12381) W/o HA, upon a full restart, checkpointing crashes

2019-05-24 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847466#comment-16847466
 ] 

Konstantin Knauf commented on FLINK-12381:
--

Thanks [~till.rohrmann] for the comments.  Overall, we end up with following 
behavior for the {{JobID}} in the {{StandaloneClusterEntryPoint}} by ascending 
precedence:

* 0 (HA), random (non HA)
*  {{--job-id}}

I will do this in a first step and will in parallel look into how to 
automatically set the {{--job-id}} per Deployment in Kubernetes.


> W/o HA, upon a full restart, checkpointing crashes
> --
>
> Key: FLINK-12381
> URL: https://issues.apache.org/jira/browse/FLINK-12381
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.8.0
> Environment: Same as FLINK-\{12379, 12377, 12376}
>Reporter: Henrik
>Assignee: Konstantin Knauf
>Priority: Major
>
> {code:java}
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> 'gs://example_bucket/flink/checkpoints//chk-16/_metadata'
>  already exists
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.createChannel(GoogleHadoopOutputStream.java:85)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.(GoogleHadoopOutputStream.java:74)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:797)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:807)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.(FsCheckpointMetadataOutputStream.java:65)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointStorageLocation.createMetadataOutputStream(FsCheckpointStorageLocation.java:104)
>     at 
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:259)
>     at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:829)
>     ... 8 more
> {code}
> Instead, it should either just overwrite the checkpoint or fail to start the 
> job completely. Partial and undefined failure is not what should happen.
>  
> Repro:
>  # Set up a single purpose job cluster (which could use much better docs btw!)
>  # Let it run with GCS checkpointing for a while with rocksdb/gs://example
>  # Kill it
>  # Start it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Commented] (FLINK-12546) Base Docker images on `library/flink`

2019-05-24 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16847452#comment-16847452
 ] 

Konstantin Knauf commented on FLINK-12546:
--

[~yunta] [https://github.com/docker-flink/docker-flink] contains the build 
scripts for the official community maintained Flink Docker images, which are 
published as {{library/flink}}. These should be the images that Flink users 
turn to, when they need Flink Docker images.

{{flink-container}} contains a Dockerfile and tooling ({{build.sh}}) to build 
Flink docker images for Job clusters, where the user jar is already packaged 
with the image. This docker image could be based on the `library/flink` instead 
of building another slidely different Flink base image FROM alpine. 

My assumption is, that {{build.sh}} is currently used by two types of users: 

1. Flink Users, who appreciate the tooling for job-clusters
2. Flink Developers, who need Docker images based on their current working 
directory and not an official release

For users (1.) we only need the `--from-release` option and simply add the user 
jar (and optionally Hadoop) to the image. For developers (2.) we additionally 
also change the Flink version. In my opinion the {{--from-archive}} is not 
really needed, because either you want to build for a release (-> 
{{--from-release}}) or a local snapshot version (-> {{--from-local-dist}}), but 
I might of course not be aware of all types of usages of this script.




 


> Base Docker images on `library/flink`
> -
>
> Key: FLINK-12546
> URL: https://issues.apache.org/jira/browse/FLINK-12546
> Project: Flink
>  Issue Type: Improvement
>  Components: Deployment / Docker
>Affects Versions: 1.8.0
>Reporter: Konstantin Knauf
>Priority: Major
>
> Currently, Flink users turn to two different places when looking for 
> "official" community Docker images. 
> * https://github.com/docker-flink/docker-flink (Flink community maintained, 
> no official Apache releases) or https://hub.docker.com/_/flink/ 
> * The tooling and Dockerfile in the {{flink-container}} component
> While users should turn to the Flink images on Docker Hub in general, the 
> {{flink-container}} component is used by many users, because it contains some 
> tooling to build images for the {{StandaloneJobClusterEntrypoint}}. Overall, 
> this causes confusion for users and the community needs to maintain two 
> Dockerfiles, which build Flink images FROM alpine or debian.
> Therefore, I propose to change the tooling ({{build.sh}}) in 
> {{flink-container}} to have only two options: 
> a) {{from-release}}, which uses `library/flink` as base image and basically 
> only adds the user jar.  (<--- for Flink users)
> b)  {{from-local-dist}}, which also uses `library/flink` as base image but 
> replaces the flink-dist by the local flink-dist ( <--- for Flink developer)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Created] (FLINK-12546) Base Docker images on `library/flink`

2019-05-17 Thread Konstantin Knauf (JIRA)
Konstantin Knauf created FLINK-12546:


 Summary: Base Docker images on `library/flink`
 Key: FLINK-12546
 URL: https://issues.apache.org/jira/browse/FLINK-12546
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Docker
Affects Versions: 1.8.0
Reporter: Konstantin Knauf


Currently, Flink users turn to two different places when looking for "official" 
community Docker images. 

* https://github.com/docker-flink/docker-flink (Flink community maintained, no 
official Apache releases) or https://hub.docker.com/_/flink/ 
* The tooling and Dockerfile in the {{flink-container}} component

While users should turn to the Flink images on Docker Hub in general, the 
{{flink-container}} component is used by many users, because it contains some 
tooling to build images for the {{StandaloneJobClusterEntrypoint}}. Overall, 
this causes confusion for users and the community needs to maintain two 
Dockerfiles, which build Flink images FROM alpine or debian.

Therefore, I propose to change the tooling ({{build.sh}}) in 
{{flink-container}} to have only two options: 

a) {{from-release}}, which uses `library/flink` as base image and basically 
only adds the user jar.  (<--- for Flink users)

b)  {{from-local-dist}}, which also uses `library/flink` as base image but 
replaces the flink-dist by the local flink-dist ( <--- for Flink developer)








--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (FLINK-12381) W/o HA, upon a full restart, checkpointing crashes

2019-05-17 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841524#comment-16841524
 ] 

Konstantin Knauf edited comment on FLINK-12381 at 5/17/19 8:23 AM:
---

[~till.rohrmann] I am happy to take this. Let me summarize what, I think, we 
need to do based on the discussion:

1) To solve the issue at hand, we only need to make sure, that {{JobID}} s are 
per-default randomly generated in {{StandaloneJobClusterEntrypoint}}, instead 
of defaulting to {{0}}, when HA is disabled. Are we breaking any public API by 
this? As far as I know, no, and a note in the release docs suffices.

2) In the case of a Job Cluster with HA enabled my understanding is, that there 
could still be a conflict even with different {{high-availability.cluster-id}} 
s, because the {{cluster-id}} is not part of the checkpoint path (or is it?). 
It only prevents clashes in Zookeeper. So, if there are left over checkpoints 
(reatained checkpoint or job never reached terminal state) from a previous job 
with a different {{cluster-id}}, but same {{state.checkpoints.dir}} and same 
default {{JobID}} ({{0}}), there would still be an issue. Therefore, in this 
case we would like to inject the {{JobID}} instead of defaulting to {{0}}.





was (Author: knaufk):
[~till.rohrmann] I am happy to take this. Let me summarize what, I think, we 
need to do based on the discussion:

1) To solve the issue at hand, we only need to make sure, that {{JobID}} s are 
per-default randomly generated in {{StandaloneJobClusterEntrypoint}}, instead 
of defaulting to {{0}}, when HA is disabled. Are we breaking any public API by 
this? As far as I know, no, and a note in the release docs suffices.

2) In the case of a Job Cluster with HA enabled my understanding is, that there 
could still be a conflict even with different 
{{high-availability.cluster-id}}s, because the {{cluster-id}} is not part of 
the checkpoint path (or is it?). It only prevents clashes in Zookeeper. So, if 
there are left over checkpoints (reatained checkpoint or job never reached 
terminal state) from a previous job with a different {{cluster-id}}, but same 
{{state.checkpoints.dir}} and same default {{JobID}} ({{0}}), there would still 
be an issue. Therefore, in this case we would like to inject the {{JobID}} 
instead of defaulting to {{0}}.




> W/o HA, upon a full restart, checkpointing crashes
> --
>
> Key: FLINK-12381
> URL: https://issues.apache.org/jira/browse/FLINK-12381
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.8.0
> Environment: Same as FLINK-\{12379, 12377, 12376}
>Reporter: Henrik
>Assignee: Konstantin Knauf
>Priority: Major
>
> {code:java}
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> 'gs://example_bucket/flink/checkpoints//chk-16/_metadata'
>  already exists
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.createChannel(GoogleHadoopOutputStream.java:85)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.(GoogleHadoopOutputStream.java:74)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:797)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:807)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.(FsCheckpointMetadataOutputStream.java:65)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointStorageLocation.createMetadataOutputStream(FsCheckpointStorageLocation.java:104)
>     at 
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:259)
>     at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:829)
>     ... 8 more
> {code}
> Instead, it should either just overwrite the checkpoint or fail to start the 
> job completely. Partial and undefined failure is not what should happen.
>  
> Repro:
>  # Set up a single purpose job cluster (which could use much better docs btw!)
>  # Let it run with GCS checkpointing for a while with rocksdb/gs://example
>  # Kill it
>  # Start it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


[jira] [Comment Edited] (FLINK-12381) W/o HA, upon a full restart, checkpointing crashes

2019-05-17 Thread Konstantin Knauf (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841524#comment-16841524
 ] 

Konstantin Knauf edited comment on FLINK-12381 at 5/17/19 8:23 AM:
---

[~till.rohrmann] I am happy to take this. Let me summarize what, I think, we 
need to do based on the discussion:

1) To solve the issue at hand, we only need to make sure, that {{JobID}} s are 
per-default randomly generated in {{StandaloneJobClusterEntrypoint}}, instead 
of defaulting to {{0}}, when HA is disabled. Are we breaking any public API by 
this? As far as I know, no, and a note in the release docs suffices.

2) In the case of a Job Cluster with HA enabled my understanding is, that there 
could still be a conflict even with different 
{{high-availability.cluster-id}}s, because the {{cluster-id}} is not part of 
the checkpoint path (or is it?). It only prevents clashes in Zookeeper. So, if 
there are left over checkpoints (reatained checkpoint or job never reached 
terminal state) from a previous job with a different {{cluster-id}}, but same 
{{state.checkpoints.dir}} and same default {{JobID}} ({{0}}), there would still 
be an issue. Therefore, in this case we would like to inject the {{JobID}} 
instead of defaulting to {{0}}.





was (Author: knaufk):
[~till.rohrmann] I am happy to take this. Let me summarize what, I think, we 
need to do based on the discussion:

1) To solve the issue at hand, we only need to make sure, that {{JobID}}s are 
per-default randomly generated in {{StandaloneJobClusterEntrypoint}}, instead 
of defaulting to {{0}}, when HA is disabled. Are we breaking any public API by 
this? As far as I know, no, and a note in the release docs suffices.

2) In the case of a Job Cluster with HA enabled my understanding is, that there 
could still be a conflict even with different 
{{high-availability.cluster-id}}s, because the {{cluster-id}} is not part of 
the checkpoint path (or is it?). It only prevents clashes in Zookeeper. So, if 
there are left over checkpoints (reatained checkpoint or job never reached 
terminal state) from a previous job with a different {{cluster-id}}, but same 
{{state.checkpoints.dir}} and same default {{JobID}} ({{0}}), there would still 
be an issue. Therefore, in this case we would like to inject the {{JobID}} 
instead of defaulting to {{0}}.




> W/o HA, upon a full restart, checkpointing crashes
> --
>
> Key: FLINK-12381
> URL: https://issues.apache.org/jira/browse/FLINK-12381
> Project: Flink
>  Issue Type: Bug
>  Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.8.0
> Environment: Same as FLINK-\{12379, 12377, 12376}
>Reporter: Henrik
>Assignee: Konstantin Knauf
>Priority: Major
>
> {code:java}
> Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: 
> 'gs://example_bucket/flink/checkpoints//chk-16/_metadata'
>  already exists
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.createChannel(GoogleHadoopOutputStream.java:85)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopOutputStream.(GoogleHadoopOutputStream.java:74)
>     at 
> com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.create(GoogleHadoopFileSystemBase.java:797)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:929)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
>     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:807)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:141)
>     at 
> org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:37)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointMetadataOutputStream.(FsCheckpointMetadataOutputStream.java:65)
>     at 
> org.apache.flink.runtime.state.filesystem.FsCheckpointStorageLocation.createMetadataOutputStream(FsCheckpointStorageLocation.java:104)
>     at 
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:259)
>     at 
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:829)
>     ... 8 more
> {code}
> Instead, it should either just overwrite the checkpoint or fail to start the 
> job completely. Partial and undefined failure is not what should happen.
>  
> Repro:
>  # Set up a single purpose job cluster (which could use much better docs btw!)
>  # Let it run with GCS checkpointing for a while with rocksdb/gs://example
>  # Kill it
>  # Start it



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)


  1   2   >