On Thu, Aug 2, 2018 at 12:17 AM, Luca Burgazzoli <lburgazz...@gmail.com> wrote: > That’s very nice start, thx Guillaume >
Yes this is really awesome to see this prototype in action. > On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gno...@apache.org> wrote: > >> I've pushed a branch with graalvm experiments. >> https://github.com/apache/camel/tree/graalvm >> In order to build the project, you need to use the GraalVM jdk and use the >> following ~/.m2/toolchains.xml adapted to your path: >> >> <?xml version="1.0" encoding="UTF8"?> >> >> <toolchains> >> >> <toolchain> >> >> <type>jdk</type> >> >> <provides> >> >> <version>1.0.0-rc4</version> >> >> <vendor>oracle</vendor> >> >> <id>graalvm-ee-1.0.0-rc4</id> >> >> </provides> >> >> <configuration> >> >> >> >> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome> >> >> </configuration> >> >> </toolchain> >> >> </toolchains> >> >> Build and run: >> gnodet•camel*(*graalvm*)*» cd platforms/graalvm >> gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install >> ... >> gnodet•camel/platforms/graalvm*(*graalvm*)*» >> ./example/target/org.apache.camel.graalvm.main >> >> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel >> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting >> >> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is >> not in use. If using streams then its recommended to enable stream caching. >> See more details at http://camel.apache.org/stream-caching.html >> >> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of >> which 0 are started >> >> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel >> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds >> >> Guillaume >> >> Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gno...@apache.org> a >> écrit : >> >> > >> > >> > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni.ferr...@gmail.com> a >> > écrit : >> > >> >> Hi Cameleers, >> >> it seems from the comments that this "Kamel" subproject is something we >> >> want to start and I think that also the main camel core will benefit >> from >> >> the new features it will bring. >> >> >> >> I would like to donate the current "Kamel" code to Apache Camel, in >> order >> >> to have a initial brick to start from. >> >> >> >> From your reactions, the name "Kamel" seems cool and I'd like to keep >> it. >> >> However if you talk to people about the "Kamel" project, they won't >> >> understand if you're talking about "Kamel with the K" or "Camel with the >> >> C". >> >> For this reasons I propose to keep "Kamel", but also use "Camel K" >> [ˈkæməl >> >> keɪ] as friendly name when talking about it. This is in line with the >> >> repository we should create for the subproject, that needs to start with >> >> "camel-" (it will be "apache/camel-k" in that case). >> >> >> >> These days I've been experimenting different ideas with Luca Burgazzoli. >> >> In >> >> particular, there have been some concerns here on the power of a >> >> declarative DSL (a new one or the existing XML one) and we've found a >> >> strategy that will allow us to use also the Java DSL in Kamel >> >> integrations. >> >> Without adding too many details here, at integration build time we can >> run >> >> the user code in a build container and inspect the produced routes to >> get >> >> metadata for the optimizer. This is one of the first things we should do >> >> next. >> >> >> >> This possibility of running code that produces integrations opened a lot >> >> of >> >> unexpected paths, that I've written down in the project roadmap ( >> >> https://github.com/nicolaferraro/integration-operator) and summarized >> >> below. >> >> >> >> One nice feature (kudos to Luca) is that we can simplify life for Kamel >> >> users up to the point that they'll just need to write their routes on a >> >> Java file and run them with e.g.: >> >> "kamel run Routes.java" >> >> >> >> Where "kamel" is a binary we release within the project. Kubernetes >> custom >> >> resources will be used under the hood, but the "kamel" binary is a >> utility >> >> that will provide a user experience comparable to that of current >> >> serverless platforms (or even better). >> >> >> >> But there's not just that. >> >> >> >> Luca wrote this weekend a prototype for having a polyglot Camel ( >> >> https://github.com/lburgazzoli/camel-routes-loader). >> >> >> >> With that, you will be able to e.g. write integrations in groovy and use >> >> the same Kamel engine: >> >> "kamel run routes.groovy" >> >> >> >> But also JavaScript: >> >> >> >> >> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2 >> >> >> >> This is a prototype right now, but a similar approach based on GraalVM >> has >> >> a lot of potential, because users can not only use their preferred >> >> language >> >> to write routes and processors, but also bind them to functionalities >> >> available in their preferred libraries. >> >> >> >> How much this will be feasible depends on the adoption of GraalVM, but >> I'm >> >> seeing many frameworks adding metadata to make GraalVM work with >> >> reflection. We've also done some work in Camel, there are some Jiras for >> >> it >> >> and first tests made by Guillaume signal that it's something feasible, >> at >> >> least for the Camel core and a subset of components. GraalVM would be >> >> important also to reduce memory footprint and improve startup time, as >> >> already said. >> >> >> > >> > I'll continue experimenting and I'll report back. >> > I did some initial experiments leveraging some modifications I did to >> > speed up the start up time [1] and on a single route [2]. >> > There may be some limitations down the road of course, but at least it >> > shows that it's feasible. Fwiw, the experiments lead to a startup time of >> > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]). >> > >> > >> >> >> >> If the GraalVM approach works (it is working for other frameworks), >> >> instead >> >> of rewriting a subset of Camel in Go (as the original proposal >> mentions), >> >> we can just sanitize and recompile our existing codebase: this way we >> >> fully >> >> leverage the strength of Apache Camel. >> >> >> > >> > Right, if we can avoid rewriting Camel, that would be much better ! >> > >> > [1] https://issues.apache.org/jira/browse/CAMEL-12688 >> > [2] >> > >> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java >> > >> > >> >> >> >> Another important thing in the roadmap is actually how we'll integrate >> >> with >> >> current FaaS platforms. We will evaluate together if it's better to >> have a >> >> tighter integration with some of them or to leverage knative for some >> >> aspects of Kamel... >> >> >> >> But, let's get started! >> >> >> >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni.ferr...@gmail.com> ha >> scritto: >> >> >> >> It's clear to me that we need to add support for our existing XML DSL, >> >> that >> >> is powerful. But there are multiple reasons why I'd like to also "add" a >> >> limited yaml/json notation to "Kamel". >> >> >> >> The first one (and simplest) is that json/yaml is the primary encoding >> for >> >> all resources exchanged with the Kubernetes API server. I'm not saying >> >> that >> >> for this reason the Integration resource cannot contain a XML >> >> (/Java/Ballerina) section, but from a pure presentational point of view, >> >> having the possibility to write simple use cases (even a "hello world") >> in >> >> pure json/yaml is better than always requiring a mix of json and XML. >> >> >> >> The second one is simplicity. Writing a optimizer (the module that >> >> materializes the integration into running code, choosing a specific >> >> runtime >> >> and configuration) for a turing-complete language like Java is not easy: >> >> even if you manage to create a good parser, it may be able to do >> >> optimizations only if you write routes in a particular way, without >> >> complex >> >> constructs. >> >> XML is ok from this point of view. The thing is that both json/yaml and >> >> XML >> >> are just two different ways to serialize object trees, that can be then >> >> statically analyzed. >> >> The point is not XML vs json/yaml, it's more about tailoring a new >> >> minimalistic DSL to the emerging use cases vs proposing "only" our >> classic >> >> way of writing integrations. I think XML can be the "advanced" way. We >> can >> >> experiment optimizations easily with the new DSL, and enable them also >> on >> >> XML if it's worth. >> >> >> >> Scripting should be part of the spec, but I'd try to use programming >> >> languages only for processing/transformation, not for the route >> >> definition. >> >> >> >> Third one is performance. Apart from the fact that json parsers are in >> >> general said to be faster than XML parsers... Given the "fast startup" >> >> target that we want to reach, we may think e.g. to translate the new DSL >> >> into Java or Go code, then compiling it. This allows doing parsing at >> >> build >> >> time in order to avoid it on startup. This kind of improvements are much >> >> easier with a limited DSL but much more difficult with a existing >> >> fully-fledged DSL.. >> >> >> >> Nicola >> >> >> >> >> >> >> >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ullg...@gmail.com> >> >> wrote: >> >> >> >> > I also like the idea but with some comments. >> >> > >> >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this. >> >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something >> >> like >> >> > Ballerina language would be better suited ? >> >> > Also in my experience even simple integrations, that is simple real >> >> world >> >> > integration and not just hello world demos, requires >> >> > you to add one or more Java class or scripting in addition to the core >> >> > components. So for it to be useful there must be some way to add >> custom >> >> > code in some way for aggregation/enrichment strategies. >> >> > >> >> > If you go with a GO implementation I would also like some fallback for >> >> > using Java since I find it unlikely that the existing base of >> components >> >> > will be ported any time soon. >> >> > >> >> > As for the name as other I like the "Kamel" name however a being a >> >> newborn >> >> > kotlin fan I know of one pre-existing project with that name >> >> > that is close enough to allow for potential confusion. >> >> > https://github.com/topicusoverheid/kamel >> >> > >> >> > Looking forward to see the result. >> >> > >> >> > // Pontus >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hi...@hiramchirino.com> >> >> wrote: >> >> > >> >> > > Love the idea. >> >> > > >> >> > > Personally, I'd keep using the existing Camel XML DSL if possible. >> >> You >> >> > can >> >> > > still embed it in the CRD. The operator that deploys it could >> >> inspect it >> >> > > and figure whats the most optimized runtime that can support the >> DSL. >> >> > > Perhaps if it's only using the restricted set of camel components >> >> > supported >> >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses >> >> that. >> >> > > Otherwise it falls back to using camel spring boot. >> >> > > >> >> > > For that to work I think we need the different runtime >> >> implementations to >> >> > > provide a way to ask them: 'hey do you support running this camel >> >> route?' >> >> > > Not a trivial thing to respond to, it might require a build step in >> >> there >> >> > > for traditional Camel runtimes. >> >> > > >> >> > > >> >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti < >> >> > anto...@stefanutti.fr> >> >> > > wrote: >> >> > > >> >> > > > Hi Nicola, >> >> > > > >> >> > > > I love the idea. >> >> > > > >> >> > > > I just wonder whether YAML/JSON is an expressive enough format in >> >> the >> >> > > long >> >> > > > term. But as you’ve mentioned, starting simple would enable >> >> > experimenting >> >> > > > some very interesting / promising optimisations. So it seems worth >> >> > taking >> >> > > > that path, instead of trying to embed a complex DSL or the >> existing >> >> XML >> >> > > DSL >> >> > > > into the CRD. >> >> > > > >> >> > > > Definitely +1 >> >> > > > >> >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni.ferr...@gmail.com> >> >> > wrote: >> >> > > > > >> >> > > > > Hi Cameleers, >> >> > > > > it's now passed some time since I started thinking about a new >> >> > project >> >> > > > that >> >> > > > > we can begin here at Apache Camel, and I'd like to have your >> >> opinion. >> >> > > > > >> >> > > > > We've already been targeting cloud-native applications with >> Camel, >> >> > > > > especially on top of Kubernetes, that is becoming "the standard" >> >> > cloud >> >> > > > > platform. But writing a Camel integration and running it on >> >> > Kubernetes >> >> > > > > requires some effort: choosing the base platform (spring-boot, >> >> karaf, >> >> > > > > simple main?), adding health checks (actuator?), packaging a >> >> docker >> >> > > image >> >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin, >> >> helm?), >> >> > > > > publishing the image on a docker registry, then finally >> deploying >> >> the >> >> > > > > resources on a Kubernetes cluster. >> >> > > > > >> >> > > > > The resulting integration container is then far from being >> optimal >> >> > > from a >> >> > > > > resource consumption point of view: it is likely that a Camel >> >> > > Spring-Boot >> >> > > > > application will require at least 200MB of RAM and also some CPU >> >> > shares >> >> > > > > because of polling threads used by many components. >> >> > > > > >> >> > > > > In case people use a CI/CD pipeline, it will take also a long >> >> time to >> >> > > get >> >> > > > > from a code update to having a Kubernetes POD up and running. >> >> > > > > Apart from compilation and image push/pull time, also startup >> >> time is >> >> > > > often >> >> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard >> >> > limits >> >> > > > on >> >> > > > > resources, making it difficult to propose this combination for >> >> > > > "serverless >> >> > > > > integration" (this term is becoming increasingly more popular). >> >> > > > > >> >> > > > > So, my proposal is to start to investigate a "more cloud-native" >> >> > > approach >> >> > > > > to integration: *making Camel integrations first-class citizens >> in >> >> > > > > Kubernetes, and making them super fast and lightweight.* >> >> > > > > >> >> > > > > We can base the project on Kubernetes Custom Resource >> Definitions >> >> > (CRD) >> >> > > > > < >> >> > > > >> >> > > >> >> > >> >> >> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ >> >> > > > >, >> >> > > > > for example a Integration CRD and have a Kubernetes "operator" >> >> > > > > <https://coreos.com/operators/> taking care of: >> >> > > > > - Optimizing the integration that we want to run >> >> > > > > - Packaging in a container >> >> > > > > - Running it on Kubernetes >> >> > > > > - Managing its entire lifecycle >> >> > > > > >> >> > > > > A Kubernetes-native integration may look like: >> >> > > > > >> >> > > > > ------------------- >> >> > > > > kind: "Integration" >> >> > > > > apiVersion: "camel.apache.org/v1alpha1" >> >> > > > > metadata: >> >> > > > > name: "example" >> >> > > > > spec: >> >> > > > > replicas: 1 >> >> > > > > routes: >> >> > > > > - id: timer >> >> > > > > route: >> >> > > > > - type: endpoint >> >> > > > > uri: timer:tick >> >> > > > > - type: endpoint >> >> > > > > uri: log:info >> >> > > > > ------------------- >> >> > > > > >> >> > > > > For those who are not familiar with Kubernetes resources, this >> >> kind >> >> > of >> >> > > > > YAML/JSON resource definitions are really common. >> >> > > > > The example route is embedded in the Kubernetes resource >> >> declaration >> >> > > and >> >> > > > > follows a basic "flow DSL". We may start from a basic one and >> >> evolve >> >> > it >> >> > > > as >> >> > > > > new requirements arrive from the community. >> >> > > > > >> >> > > > > I've made a very simple (but working) POC here: >> >> > > > > https://github.com/nicolaferraro/integration-operator. >> >> > > > > >> >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project >> >> codename >> >> > can >> >> > > > be " >> >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of >> nice >> >> > > > features. >> >> > > > > >> >> > > > > For example, we can propose "Kamel" as "ideal" platform for >> >> > "serverless >> >> > > > > integration" (I see many people reinventing the wheel out >> there): >> >> the >> >> > > > > operator can reduce resource consumption of a single integration >> >> by >> >> > > > > optimizing the runtime and also pause/resume integrations when >> >> they >> >> > are >> >> > > > not >> >> > > > > used, that is the basic idea behind "serverless" (e.g. think to >> >> > > > > HTTP-triggered integrations, but not only). >> >> > > > > Focusing on serverless will bring more emphasis on push-based >> >> > > > notifications >> >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are >> >> rarely >> >> > > > used in >> >> > > > > Camel components, that prefer a poll based approach being it >> >> simpler >> >> > to >> >> > > > use >> >> > > > > in classic deployments, but not so good in the cloud, where more >> >> > > > resources >> >> > > > > become higher direct costs for the users. >> >> > > > > >> >> > > > > The presence of the simplified DSL enables also experimenting on >> >> > > > "*reduced* >> >> > > > > subsets of Camel" implemented in languages other than Java, for >> >> > example >> >> > > > one >> >> > > > > language that has a reactive approach on thread scheduling and a >> >> > really >> >> > > > low >> >> > > > > memory footprint, like Go. >> >> > > > > >> >> > > > > But apart from this kind of experiments (that are valid IMO), >> the >> >> > > "Kamel" >> >> > > > > optimizer will have free room to choose the right platform for >> the >> >> > > > > integration that the user wants to run, including, in the >> future, >> >> > doing >> >> > > > AOT >> >> > > > > compilation using Graal/VM (less memory, faster startup) if the >> >> > > features >> >> > > > > (components) used in the integration are supporting it (maybe we >> >> can >> >> > > add >> >> > > > > AOT compilation in the roadmap for Camel 3). >> >> > > > > A silly optimization: integrations starting from "timer:..." may >> >> be >> >> > > > > scheduled directly with Kubernetes CronJobs, so they will >> consume >> >> > > > resources >> >> > > > > only when actually running. >> >> > > > > >> >> > > > > Being the final integrations lightweight and being the DSL >> >> > > > > language-independent, we may see a increased adoption of Camel >> >> also >> >> > as >> >> > > > > agile integration layer for not-only-java applications (both >> >> "cloud" >> >> > > and >> >> > > > > "serverless" applications). >> >> > > > > >> >> > > > > I'm the first one that would like to work on a project ilke >> this. >> >> > I've >> >> > > > > worked on many Kubernetes/Openshift based applications and >> >> frameworks >> >> > > in >> >> > > > > the past years, also on operators and CRDs, and I think this way >> >> of >> >> > > > > redesigning integrations has a lot of potential. >> >> > > > > >> >> > > > > Integrations will not be necessarily limited to the simplified >> >> DSL, >> >> > but >> >> > > > we >> >> > > > > can add extension points for scripting and even custom libraries >> >> > > > (although >> >> > > > > limiting the freedom of the optimizer). >> >> > > > > >> >> > > > > The most important thing: it may become a great project, since >> >> it's >> >> > > > driven >> >> > > > > by a great community. >> >> > > > > >> >> > > > > So, what do you think? Is it crazy enough? >> >> > > > > >> >> > > > > Nicola >> >> > > > >> >> > > > >> >> > > >> >> > > -- >> >> > > Hiram Chirino >> >> > > Engineering | Red Hat, Inc. >> >> > > hchir...@redhat.com | fusesource.com | redhat.com >> >> > > skype: hiramchirino | twitter: @hiramchirino >> >> > > >> >> > >> >> >> > >> > >> > -- >> > ------------------------ >> > Guillaume Nodet >> > >> > >> >> -- >> ------------------------ >> Guillaume Nodet >> > -- > -- > Luca Burgazzoli -- Claus Ibsen ----------------- http://davsclaus.com @davsclaus Camel in Action 2: https://www.manning.com/ibsen2
