Hi! Thanks for joining the discussion a lot!
By creating new templates folder in the Beam we try to make templates generic. As it seems to us, another important reason for adding a new folder is Gradle build. We will configure Gradle build directly for templates, without a necessity to build all examples, and without unused dependencies. Creating subfolders for different runners makes a sense, however, the proposed template could be running on the different runners, and GCP here is an optional approach than a requirement. I suggest adding a subfolder for different languages, under the /templates e.g.: /templates/python, /templates/java, etc. If we will need more differences between templates we can add new subfolders for specific cases in the future. Thank you, Ilya On 15 Oct 2020, 07:30 +0300, Reza Ardeshir Rokni <[email protected]>, wrote: Just a thought, but what if in the future there were templates for other runners? Then having a template folder would fit nicely no? We could even have a runner specific subfolder and maybe even a shared area for things that could be used by all templates for all runners? On Thu, 15 Oct 2020 at 11:47, Kenneth Knowles <[email protected]<mailto:[email protected]>> wrote: Hi Ilya, I have added you to the "Contributors" role on Jira so you can be assigned tickets, and given you the ticket you filed since you are already solving it. Thanks! I have a very high level thought: Since Dataflow's "Flex Templates" feature is just any pipeline, perhaps the main pipeline can be more of an "example" and fit into the `examples/` folder? Then the containerization and Google-specific* JSON could be alongside. In this way, users of other runners could possibly use or learn from it even if they are not interested in GCP. I understand this is not your primary goal, considering the contribution. I just want to open this for discussion. Kenn *In fact, the JSON is very generic. It is not really "Google specific" in concept, just in practice. On Wed, Oct 14, 2020 at 12:14 PM Ilya Kozyrev <[email protected]<mailto:[email protected]>> wrote: Hi Beam Community, There was no feedback on the proposal, and I would like to submit PR for this proposal. I created a JIRA improvement<https://issues.apache.org/jira/browse/BEAM-11065> to track this proposal and now submitting PR<https://github.com/apache/beam/pull/13112> in the Beam repository related to the proposal that I sent before. We suggest adding /template folder to the repository root to help discover templates by developers. This will provide structure for future templates development for Beam. Could someone kindle help with reviewing the PR<https://github.com/apache/beam/pull/13112> ? Thank you, Ilya On 7 Oct 2020, at 21:23, Ilya Kozyrev <[email protected]<mailto:[email protected]>> wrote: Hi Beam Community, I have a proposal to add Apache Beam example that is a template to ingest data from Apache Kafka to Google Cloud Pub/Sub. More detailed information about the proposed template can be found in README<https://github.com/akvelon/beam/blob/KafkaToPubsubTemplate/templates/README.md> file, and a prototype<https://github.com/akvelon/beam/pull/1> was built with a team. I'd like to ask for your feedback before moving forward with finishing the template. I did not see a folder that provides easily discoverable templates to a developer. I would like to propose adding a "templates" folder where other Apache Beam templates may be added in the future. E.g., beam/templates/java/kafka-to-pubsub could be used for the Kafka to Pub/Sub template. Please share your feedback/comments about this proposal in the thread. Thank you, Ilya
