Hi Kobe, Thank you for sharing your feedback, experience and ideas! They described examples catalog improvements with great details that’s helpful for our planning!
There are now few options with Docker environments for Beam: * Apache Beam development environment Docker: start-build-env.sh<https://github.com/apache/beam/blob/master/start-build-env.sh> in Beam’s repository creates Docker based local environment. It’s mentioned on https://beam.apache.org/contribute/ - and may be would be helpful to also note it additionally somewhere in Get Started on Beam website * Docker runtime environments: https://beam.apache.org/documentation/runtime/environments/ Hope this would be helpful with local environment for experimentation scenarios. Thanks, Alex From: Phx Feng <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Monday, January 24, 2022 at 6:31 PM To: "[email protected]" <[email protected]> Cc: Pablo Estrada <[email protected]>, Damon Douglas <[email protected]>, Aizhamal Nurmamat kyzy <[email protected]>, Artur Khanin <[email protected]>, Ilya Kozyrev <[email protected]> Subject: [EXTERNAL] Re: Try Beam Playground (Beta) - Interactive Learning for Apache Beam This is Cool, thanks for sharing and the great efforts. Sorry for being ignorant to reply immediately after I saw it. I have been disconnected with apache beam for a long time and things that gave me headache before were: 1. higher bar for streaming dev compared with batch for end user in fine grit level control. 2. how much benefits for using apache beam over flink and spark but introduce performance overhead and not reduce infra maintenance. 3. what cases would fit for streaming and when to use it. Let user experiment and engage with examples easily will definitely help more adoptions by data engineers and scientists. First glance, I was thinking there shall be one setup tutorial with corresponding docker images for people who would rather initiate such env in a company for experimenting with internal datasets based on examples if possible (security concern). If end user can't prove existing use case in experimentation phase, they would never try that by themselves and no migration happen, then may no public cloud chance ^_^ Then, I was expecting more IOs by using different connectors when I expanded the IO tag, since some connectors require people knowing the details and help integration e.g. event partitioning and file shard for hdfs. and experimentation phase could consider minicluster or standalone mode locally for external component before data model review and the submission mode. Also I'm wondering about sorting the examples from easy to hard based on scenarios, then a fresh learner would not need to know how many functions beam organized or go to the tutorial for all functions first, and be more business oriented first. like count features for scientist models, user reaction analysis from different channels, etc. Last, I'm not checking any streaming input there, and not sure the output will be rolling displayed (back pressure and rate limit will protect workload), which is cool and easy to check outputs from all stages in a single pipeline to know how data changed, and also the watermark visutationlization for different stages and how many events late arrived on that stage which helps delay events overview and adjust the trigger strategy, and this is cool too. Forgive me if any inappropriate point, thank you! Kobe On Mon, Jan 24, 2022 at 3:34 PM Alex Kosolapov <[email protected]<mailto:[email protected]>> wrote: TL:DR: We invite you to try out Apache Beam Playground (Beta)<https://frontend-beta-dot-apache-beam-testing.appspot.com/> [1] and share your feedback using built-in “Enjoying Playground?” or feedback form<https://forms.gle/Yt3D9tz9zEpeb97E9> [2]. Hi all, We are inviting you to try Beam Playground (Beta)<https://frontend-beta-dot-apache-beam-testing.appspot.com/> [1] - an interactive environment to try out Apache Beam transforms and examples without having to set up a Beam environment (link to RFC on proposal<https://lists.apache.org/thread/swlct79fbrkztr9o7mp6d0swccn6mztp> [3]). Key features: * Discover Beam examples by browsing or searching examples catalog sourced from Apache Beam Github * Select Java, Python or Go SDK to execute the example in Beam direct runner * Sourcing examples based on Playground tag from Apache Beam Github repository using CI/CD. * Responsive UX with code highlighting, flexible layout, color schemes, and other features for desktop browsers * Embedding a Playground example on Apache Beam web site that prompts a user to Try in Playground - e.g., Playground Quickstart page [4] https://beam.apache.org/get-started/try-beam-playground/ Known issues: * Link to known issues<https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20beam-playground%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC> [5] Highlights of some features that we plan to build and now in progress: * SCIO SDK support * Display Graph for Java, Python SDKs * Display “Quickstart” examples first in the catalog * Embedding Playground example to support running example and displaying output in place (as iframe on a web page) * Persist light/dark color scheme selection Please share your feedback using the built-in “Enjoying Playground?”, feedback form<https://forms.gle/Yt3D9tz9zEpeb97E9> or reply to this email. Thank you for trying this Beam Playground Beta out! Thanks, Beam Playground team === [1] Apache Beam Playground (Beta) https://frontend-beta-dot-apache-beam-testing.appspot.com/ [2] Feedback form https://forms.gle/Yt3D9tz9zEpeb97E9 [3] [RFC][design/idea] Beam Playground - Interactive Learning for Apache Beam https://lists.apache.org/thread/swlct79fbrkztr9o7mp6d0swccn6mztp [4] Playground Quickstart page https://beam.apache.org/get-started/try-beam-playground/ [5] Known issues https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20issuetype%20%3D%20Bug%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20beam-playground%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC -- Yours Sincerely Kobe Feng
