This is a wonderful idea and I believe we can improve our community with a subproject like this.
So +1 for me. -- Andrea Cosentino ---------------------------------- Apache Camel PMC Chair Apache Karaf Committer Apache Servicemix PMC Member Email: ancosen1...@yahoo.com Twitter: @oscerd2 Github: oscerd On Monday, July 30, 2018, 5:49:50 PM GMT+2, Nicola Ferraro <ni.ferr...@gmail.com> wrote: Hi Cameleers, it seems from the comments that this "Kamel" subproject is something we want to start and I think that also the main camel core will benefit from the new features it will bring. I would like to donate the current "Kamel" code to Apache Camel, in order to have a initial brick to start from. >From your reactions, the name "Kamel" seems cool and I'd like to keep it. However if you talk to people about the "Kamel" project, they won't understand if you're talking about "Kamel with the K" or "Camel with the C". For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl keɪ] as friendly name when talking about it. This is in line with the repository we should create for the subproject, that needs to start with "camel-" (it will be "apache/camel-k" in that case). These days I've been experimenting different ideas with Luca Burgazzoli. In particular, there have been some concerns here on the power of a declarative DSL (a new one or the existing XML one) and we've found a strategy that will allow us to use also the Java DSL in Kamel integrations. Without adding too many details here, at integration build time we can run the user code in a build container and inspect the produced routes to get metadata for the optimizer. This is one of the first things we should do next. This possibility of running code that produces integrations opened a lot of unexpected paths, that I've written down in the project roadmap ( https://github.com/nicolaferraro/integration-operator) and summarized below. One nice feature (kudos to Luca) is that we can simplify life for Kamel users up to the point that they'll just need to write their routes on a Java file and run them with e.g.: "kamel run Routes.java" Where "kamel" is a binary we release within the project. Kubernetes custom resources will be used under the hood, but the "kamel" binary is a utility that will provide a user experience comparable to that of current serverless platforms (or even better). But there's not just that. Luca wrote this weekend a prototype for having a polyglot Camel ( https://github.com/lburgazzoli/camel-routes-loader). With that, you will be able to e.g. write integrations in groovy and use the same Kamel engine: "kamel run routes.groovy" But also JavaScript: https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2 This is a prototype right now, but a similar approach based on GraalVM has a lot of potential, because users can not only use their preferred language to write routes and processors, but also bind them to functionalities available in their preferred libraries. How much this will be feasible depends on the adoption of GraalVM, but I'm seeing many frameworks adding metadata to make GraalVM work with reflection. We've also done some work in Camel, there are some Jiras for it and first tests made by Guillaume signal that it's something feasible, at least for the Camel core and a subset of components. GraalVM would be important also to reduce memory footprint and improve startup time, as already said. If the GraalVM approach works (it is working for other frameworks), instead of rewriting a subset of Camel in Go (as the original proposal mentions), we can just sanitize and recompile our existing codebase: this way we fully leverage the strength of Apache Camel. Another important thing in the roadmap is actually how we'll integrate with current FaaS platforms. We will evaluate together if it's better to have a tighter integration with some of them or to leverage knative for some aspects of Kamel... But, let's get started! Il 19 lug 2018 10:45, "Nicola Ferraro" <ni.ferr...@gmail.com> ha scritto: It's clear to me that we need to add support for our existing XML DSL, that is powerful. But there are multiple reasons why I'd like to also "add" a limited yaml/json notation to "Kamel". The first one (and simplest) is that json/yaml is the primary encoding for all resources exchanged with the Kubernetes API server. I'm not saying that for this reason the Integration resource cannot contain a XML (/Java/Ballerina) section, but from a pure presentational point of view, having the possibility to write simple use cases (even a "hello world") in pure json/yaml is better than always requiring a mix of json and XML. The second one is simplicity. Writing a optimizer (the module that materializes the integration into running code, choosing a specific runtime and configuration) for a turing-complete language like Java is not easy: even if you manage to create a good parser, it may be able to do optimizations only if you write routes in a particular way, without complex constructs. XML is ok from this point of view. The thing is that both json/yaml and XML are just two different ways to serialize object trees, that can be then statically analyzed. The point is not XML vs json/yaml, it's more about tailoring a new minimalistic DSL to the emerging use cases vs proposing "only" our classic way of writing integrations. I think XML can be the "advanced" way. We can experiment optimizations easily with the new DSL, and enable them also on XML if it's worth. Scripting should be part of the spec, but I'd try to use programming languages only for processing/transformation, not for the route definition. Third one is performance. Apart from the fact that json parsers are in general said to be faster than XML parsers... Given the "fast startup" target that we want to reach, we may think e.g. to translate the new DSL into Java or Go code, then compiling it. This allows doing parsing at build time in order to avoid it on startup. This kind of improvements are much easier with a limited DSL but much more difficult with a existing fully-fledged DSL.. Nicola On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ullg...@gmail.com> wrote: > I also like the idea but with some comments. > > As Hiram Chirino I'm not sure YAML/JSON is the best language for this. > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like > Ballerina language would be better suited ? > Also in my experience even simple integrations, that is simple real world > integration and not just hello world demos, requires > you to add one or more Java class or scripting in addition to the core > components. So for it to be useful there must be some way to add custom > code in some way for aggregation/enrichment strategies. > > If you go with a GO implementation I would also like some fallback for > using Java since I find it unlikely that the existing base of components > will be ported any time soon. > > As for the name as other I like the "Kamel" name however a being a newborn > kotlin fan I know of one pre-existing project with that name > that is close enough to allow for potential confusion. > https://github.com/topicusoverheid/kamel > > Looking forward to see the result. > > // Pontus > > > > > > > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> wrote: > > > Love the idea. > > > > Personally, I'd keep using the existing Camel XML DSL if possible. You > can > > still embed it in the CRD. The operator that deploys it could inspect it > > and figure whats the most optimized runtime that can support the DSL. > > Perhaps if it's only using the restricted set of camel components > supported > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that. > > Otherwise it falls back to using camel spring boot. > > > > For that to work I think we need the different runtime implementations to > > provide a way to ask them: 'hey do you support running this camel route?' > > Not a trivial thing to respond to, it might require a build step in there > > for traditional Camel runtimes. > > > > > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti < > anto...@stefanutti.fr> > > wrote: > > > > > Hi Nicola, > > > > > > I love the idea. > > > > > > I just wonder whether YAML/JSON is an expressive enough format in the > > long > > > term. But as you’ve mentioned, starting simple would enable > experimenting > > > some very interesting / promising optimisations. So it seems worth > taking > > > that path, instead of trying to embed a complex DSL or the existing XML > > DSL > > > into the CRD. > > > > > > Definitely +1 > > > > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni.ferr...@gmail.com> > wrote: > > > > > > > > Hi Cameleers, > > > > it's now passed some time since I started thinking about a new > project > > > that > > > > we can begin here at Apache Camel, and I'd like to have your opinion. > > > > > > > > We've already been targeting cloud-native applications with Camel, > > > > especially on top of Kubernetes, that is becoming "the standard" > cloud > > > > platform. But writing a Camel integration and running it on > Kubernetes > > > > requires some effort: choosing the base platform (spring-boot, karaf, > > > > simple main?), adding health checks (actuator?), packaging a docker > > image > > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?), > > > > publishing the image on a docker registry, then finally deploying the > > > > resources on a Kubernetes cluster. > > > > > > > > The resulting integration container is then far from being optimal > > from a > > > > resource consumption point of view: it is likely that a Camel > > Spring-Boot > > > > application will require at least 200MB of RAM and also some CPU > shares > > > > because of polling threads used by many components. > > > > > > > > In case people use a CI/CD pipeline, it will take also a long time to > > get > > > > from a code update to having a Kubernetes POD up and running. > > > > Apart from compilation and image push/pull time, also startup time is > > > often > > > > ~10 seconds for Camel + Spring-Boot in a container with standard > limits > > > on > > > > resources, making it difficult to propose this combination for > > > "serverless > > > > integration" (this term is becoming increasingly more popular). > > > > > > > > So, my proposal is to start to investigate a "more cloud-native" > > approach > > > > to integration: *making Camel integrations first-class citizens in > > > > Kubernetes, and making them super fast and lightweight.* > > > > > > > > We can base the project on Kubernetes Custom Resource Definitions > (CRD) > > > > < > > > > > > https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ > > > >, > > > > for example a Integration CRD and have a Kubernetes "operator" > > > > <https://coreos.com/operators/> taking care of: > > > > - Optimizing the integration that we want to run > > > > - Packaging in a container > > > > - Running it on Kubernetes > > > > - Managing its entire lifecycle > > > > > > > > A Kubernetes-native integration may look like: > > > > > > > > ------------------- > > > > kind: "Integration" > > > > apiVersion: "camel.apache.org/v1alpha1" > > > > metadata: > > > > name: "example" > > > > spec: > > > > replicas: 1 > > > > routes: > > > > - id: timer > > > > route: > > > > - type: endpoint > > > > uri: timer:tick > > > > - type: endpoint > > > > uri: log:info > > > > ------------------- > > > > > > > > For those who are not familiar with Kubernetes resources, this kind > of > > > > YAML/JSON resource definitions are really common. > > > > The example route is embedded in the Kubernetes resource declaration > > and > > > > follows a basic "flow DSL". We may start from a basic one and evolve > it > > > as > > > > new requirements arrive from the community. > > > > > > > > I've made a very simple (but working) POC here: > > > > https://github.com/nicolaferraro/integration-operator. > > > > > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename > can > > > be " > > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice > > > features. > > > > > > > > For example, we can propose "Kamel" as "ideal" platform for > "serverless > > > > integration" (I see many people reinventing the wheel out there): the > > > > operator can reduce resource consumption of a single integration by > > > > optimizing the runtime and also pause/resume integrations when they > are > > > not > > > > used, that is the basic idea behind "serverless" (e.g. think to > > > > HTTP-triggered integrations, but not only). > > > > Focusing on serverless will bring more emphasis on push-based > > > notifications > > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely > > > used in > > > > Camel components, that prefer a poll based approach being it simpler > to > > > use > > > > in classic deployments, but not so good in the cloud, where more > > > resources > > > > become higher direct costs for the users. > > > > > > > > The presence of the simplified DSL enables also experimenting on > > > "*reduced* > > > > subsets of Camel" implemented in languages other than Java, for > example > > > one > > > > language that has a reactive approach on thread scheduling and a > really > > > low > > > > memory footprint, like Go. > > > > > > > > But apart from this kind of experiments (that are valid IMO), the > > "Kamel" > > > > optimizer will have free room to choose the right platform for the > > > > integration that the user wants to run, including, in the future, > doing > > > AOT > > > > compilation using Graal/VM (less memory, faster startup) if the > > features > > > > (components) used in the integration are supporting it (maybe we can > > add > > > > AOT compilation in the roadmap for Camel 3). > > > > A silly optimization: integrations starting from "timer:..." may be > > > > scheduled directly with Kubernetes CronJobs, so they will consume > > > resources > > > > only when actually running. > > > > > > > > Being the final integrations lightweight and being the DSL > > > > language-independent, we may see a increased adoption of Camel also > as > > > > agile integration layer for not-only-java applications (both "cloud" > > and > > > > "serverless" applications). > > > > > > > > I'm the first one that would like to work on a project ilke this. > I've > > > > worked on many Kubernetes/Openshift based applications and frameworks > > in > > > > the past years, also on operators and CRDs, and I think this way of > > > > redesigning integrations has a lot of potential. > > > > > > > > Integrations will not be necessarily limited to the simplified DSL, > but > > > we > > > > can add extension points for scripting and even custom libraries > > > (although > > > > limiting the freedom of the optimizer). > > > > > > > > The most important thing: it may become a great project, since it's > > > driven > > > > by a great community. > > > > > > > > So, what do you think? Is it crazy enough? > > > > > > > > Nicola > > > > > > > > > > -- > > Hiram Chirino > > Engineering | Red Hat, Inc. > > hchir...@redhat.com | fusesource.com | redhat.com > > skype: hiramchirino | twitter: @hiramchirino > > >
