[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924205#comment-16924205
]
Fabian Hueske commented on FLINK-12749:
---
Playground job code removed from master (1.10) with
8998b2fe286d4ac788011f03fc6953f8b040b119
Playground readded to master with a17623ba51f7cae83bc789cd4f8ffc7f105a8715
added to release-1.9 with cd756d6609064254cfe50191635ea762af477793
added to release-1.8 with 9868ae1dde0ac6709b4b7a8afa6e7121cfb72b4d
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16910513#comment-16910513
]
Fabian Hueske commented on FLINK-12749:
---
After some discussions on the dev mailing list
([link-1|https://lists.apache.org/thread.html/d7071dadbb266d46f593a96bb461244a12cf6ea4ea08275b028da312@%3Cdev.flink.apache.org%3E],
[link-2|[https://lists.apache.org/thread.html/4f54c0b4162e3db8626afdca5c354050282282d3cc229d01f2d8ca3e@%3Cdev.flink.apache.org%3E]|https://lists.apache.org/thread.html/4f54c0b4162e3db8626afdca5c354050282282d3cc229d01f2d8ca3e@%3Cdev.flink.apache.org%3E],]),
we decided to move the Java code of the demo job to the {{flink-playgrounds}}
repository. This also includes an environment to build a custom Docker image.
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882682#comment-16882682
]
Robert Metzger commented on FLINK-12749:
I'm happy to create an Apache repo for this, but we should probably first
discuss this on the dev@ list.
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882446#comment-16882446
]
Seth Wiesman commented on FLINK-12749:
--
+1 for option 2. It will be the best user experience.
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882185#comment-16882185
]
Fabian Hueske commented on FLINK-12749:
---
I'm also in favor of option 2 because it is more convenient for users.
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882176#comment-16882176
]
Robert Metzger commented on FLINK-12749:
I prefer option 2.
Option 1 will be at risk of being outdated quite quickly. The experience for
both users and developers will be poor
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (FLINK-12749) Getting Started - Docker Playgrounds - Flink Cluster Playground
[
https://issues.apache.org/jira/browse/FLINK-12749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882170#comment-16882170
]
Konstantin Knauf commented on FLINK-12749:
--
In order to spin up the playground the users need a few files on their machine,
namely the following:
{noformat}
playground/
├── docker-compose.yaml
└── conf
└── flink-conf.yaml
└── log4j-cli.properties
└── log4j-console.properties
{noformat}
The conf directory will eventually not be necessary anymore, while the
docker-compose configuration is of course a prerequisite. I see two options to
publish these configurations files:
*Option 1: Embedded in the Documentation*
The necessary files can be part of the documentation (expandable) and new users
create the directory structure by copying the content from there.
Pro:
* Correct Flink Version is automatically populated.
* Playground is only maintained at one place, not additional dependencies
(besides `library/flink` image and `StateMachineExample`)
Contra:
* cumbersome and error-prone for the user
*Option 2: Dedicated Repository for Playground Setup*
We create a new Apache Flink repository `apache/flink-playgrounds` with the
following directory structure:
{noformat}
flink-playground
└──flink-cluster-playground/
├── docker-compose.yaml
└── conf
└── flink-conf.yaml
└── log4j-cli.properties
└── log4j-console.properties
└──interactive-sql-playground/
{noformat}
The `interactive-sql-playground` will contain whatever is neccessry for
FLINK-12750.
We could have tags for each Flink version, so that we can automatically link to
the correct tag in the `flink-playgrounds` repository from the respective
documentation version.
Pro:
* user only needs to checkout this small repository and is ready to go.
* we can make sure the playground setup always works, because we explicitly
bump the Flink version once the new images are ready and we have tested the
playground
Contra:
* additional repository to maintain next to the documentation
Any opinions? [~fhueske] [~rmetzger] [~sjwiesman]
> Getting Started - Docker Playgrounds - Flink Cluster Playground
> ---
>
> Key: FLINK-12749
> URL: https://issues.apache.org/jira/browse/FLINK-12749
> Project: Flink
> Issue Type: Sub-task
> Components: Documentation
>Reporter: Konstantin Knauf
>Assignee: Konstantin Knauf
>Priority: Major
>
> The planned structure for the new Getting Started Guide is
> * Flink Overview (~ two pages)
> * Project Setup
> ** Java
> ** Scala
> ** Python
> * Quickstarts
> ** Example Walkthrough - Table API / SQL
> ** Example Walkthrough - DataStream API
> * Docker Playgrounds
> ** Flink Cluster Playground
> ** Flink Interactive SQL Playground
> In this ticket we add the Flink Cluster Playground, a docker-compose based
> setup consisting of Apache Kafka and Apache Flink (Flink Session Cluster),
> including a step-by-step guide for some common commands (job submission,
> savepoints, etc).
> *Some Open Questions:*
> * Which Flink images to use? `library/flink` with dynamic properties would be
> the most maintainable, I think. It would be preferable, if we don't need to
> host any custom images for this, but can rely on the existing plain Flink
> images.
> * Which Flink jobs to use? An updated version
> {{org.apache.flink.streaming.examples.statemachine.StateMachineExample}}
> might be a good option as it can with or without Kafka and contains a data
> generator writing to Kafka already (see next questions).
> * How to get data into Kafka? Maybe just provide a small bash
> script/one-liner to produce into Kafka topic or see question above.
> * Which Kafka Images to use? https://hub.docker.com/r/wurstmeister/kafka/
> seems to be well-maintained and is openly available.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
