[jira] [Comment Edited] (KAFKA-8629) Kafka Streams Apps to support small native images through GraalVM
[ https://issues.apache.org/jira/browse/KAFKA-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883067#comment-16883067 ] Staffan Olsson edited comment on KAFKA-8629 at 7/11/19 3:08 PM: Quarkus has a [Streams extension|https://quarkus.io/guides/kafka-streams-guide]. For details see [https://github.com/quarkusio/quarkus/pull/2693], [https://github.com/quarkusio/quarkus/issues/2863] and [https://github.com/quarkusio/quarkus/pull/2953.] Note also that IMO there's a blocker with Snappy and LZ4 disabled: [https://github.com/quarkusio/quarkus/issues/2718]. I've done some work on native builds of topic admin CLI tools, where the rewards in terms of reduced startup times really pay off: [https://github.com/solsson/dockerfiles/pull/25] ([example usage|https://github.com/Yolean/kubernetes-kafka/commit/af63569c41b78195b85c0ff7f919df93b2ef6003]). I've only used these builds in testing, but seen no regressions yet. They reduce CPU consumption for automated (by shell script) admin tasks by 90% and memory consumption by 80%. I did this work before I discovered [https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md] which would have made things a lot easier :) was (Author: solsson): Quarkus has a [Streams extension|[https://quarkus.io/guides/kafka-streams-guide]]. For details see [https://github.com/quarkusio/quarkus/pull/2693], [https://github.com/quarkusio/quarkus/issues/2863] and [https://github.com/quarkusio/quarkus/pull/2953.] Note also that IMO there's a blocker with Snappy and LZ4 disabled: [https://github.com/quarkusio/quarkus/issues/2718]. I've done some work on native builds of topic admin CLI tools, where the rewards in terms of reduced startup times really pay off: [https://github.com/solsson/dockerfiles/pull/25] ([example usage|https://github.com/Yolean/kubernetes-kafka/commit/af63569c41b78195b85c0ff7f919df93b2ef6003]). I've only used these builds in testing, but seen no regressions yet. They reduce CPU consumption for automated (by shell script) admin tasks by 90% and memory consumption by 80%. I did this work before I discovered [https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md] which would have made things a lot easier :) > Kafka Streams Apps to support small native images through GraalVM > - > > Key: KAFKA-8629 > URL: https://issues.apache.org/jira/browse/KAFKA-8629 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 > Environment: OSX > Linux on Docker >Reporter: Andy Muir >Priority: Minor > > I'm investigating using [GraalVM|http://example.com/] to help with reducing > docker image size and required resources for a simple Kafka Streams > microservice. To this end, I'm looking at running a microservice which: > 1) consumes from a Kafka topic (XML) > 2) Transforms into JSON > 3) Produces to a new Kafka topic. > The Kafka Streams app running in the JVM works fine. > When I attempt to build it to a GraalVM native image (binary executable which > does not require the JVM, hence smaller image size and less resources), I > encountered a few > [incompatibilities|https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md] > with the source code in Kafka. > I've implemented a workaround for each of these in a fork (link to follow) to > help establish if it is feasible. I don't intend (at this stage) for the > changes to be applied to the broker - I'm only after Kafka Streams for now. > I'm not sure whether it'd be a good idea for the broker itself to run as a > native image! > There were 2 issues getting the native image with kafka streams: > 1) Some Reflection use cases using MethodHandle > 2) Anything JMX > To work around these issues, I have: > 1) Replaced use of MethodHandle with alternatives > 2) Commented out the JMX code in a few places > While the first may be sustainable, I'd expect that the 2nd option should be > put behind a configuration switch to allow the existing code to be used by > default and turning off JMX if configured. > *I haven't created a PR for now, as I'd like feedback to decide if it is > going to be feasible to carry this forwards.* -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (KAFKA-8629) Kafka Streams Apps to support small native images through GraalVM
[ https://issues.apache.org/jira/browse/KAFKA-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883067#comment-16883067 ] Staffan Olsson edited comment on KAFKA-8629 at 7/11/19 3:08 PM: Quarkus has a [Streams extension|[https://quarkus.io/guides/kafka-streams-guide]]. For details see [https://github.com/quarkusio/quarkus/pull/2693], [https://github.com/quarkusio/quarkus/issues/2863] and [https://github.com/quarkusio/quarkus/pull/2953.] Note also that IMO there's a blocker with Snappy and LZ4 disabled: [https://github.com/quarkusio/quarkus/issues/2718]. I've done some work on native builds of topic admin CLI tools, where the rewards in terms of reduced startup times really pay off: [https://github.com/solsson/dockerfiles/pull/25] ([example usage|https://github.com/Yolean/kubernetes-kafka/commit/af63569c41b78195b85c0ff7f919df93b2ef6003]). I've only used these builds in testing, but seen no regressions yet. They reduce CPU consumption for automated (by shell script) admin tasks by 90% and memory consumption by 80%. I did this work before I discovered [https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md] which would have made things a lot easier :) was (Author: solsson): Quarkus has a [Streams extension|[https://quarkus.io/guides/kafka-streams-guide]|https://quarkus.io/guides/kafka-streams-guide]. For details see [https://github.com/quarkusio/quarkus/pull/2693], [https://github.com/quarkusio/quarkus/issues/2863] and [https://github.com/quarkusio/quarkus/pull/2953.] Note also that IMO there's a blocker with Snappy and LZ4 disabled: [https://github.com/quarkusio/quarkus/issues/2718]. I've done some work on native builds of topic admin CLI tools, where the rewards in terms of reduced startup times really pay off: [https://github.com/solsson/dockerfiles/pull/25] ([example usage|https://github.com/Yolean/kubernetes-kafka/commit/af63569c41b78195b85c0ff7f919df93b2ef6003]). I've only used these builds in testing, but seen no regressions yet. They reduce CPU consumption for automated (by shell script) admin tasks by 90% and memory consumption by 80%. I did this work before I discovered [https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md] which would have made things a lot easier :) > Kafka Streams Apps to support small native images through GraalVM > - > > Key: KAFKA-8629 > URL: https://issues.apache.org/jira/browse/KAFKA-8629 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 > Environment: OSX > Linux on Docker >Reporter: Andy Muir >Priority: Minor > > I'm investigating using [GraalVM|http://example.com/] to help with reducing > docker image size and required resources for a simple Kafka Streams > microservice. To this end, I'm looking at running a microservice which: > 1) consumes from a Kafka topic (XML) > 2) Transforms into JSON > 3) Produces to a new Kafka topic. > The Kafka Streams app running in the JVM works fine. > When I attempt to build it to a GraalVM native image (binary executable which > does not require the JVM, hence smaller image size and less resources), I > encountered a few > [incompatibilities|https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md] > with the source code in Kafka. > I've implemented a workaround for each of these in a fork (link to follow) to > help establish if it is feasible. I don't intend (at this stage) for the > changes to be applied to the broker - I'm only after Kafka Streams for now. > I'm not sure whether it'd be a good idea for the broker itself to run as a > native image! > There were 2 issues getting the native image with kafka streams: > 1) Some Reflection use cases using MethodHandle > 2) Anything JMX > To work around these issues, I have: > 1) Replaced use of MethodHandle with alternatives > 2) Commented out the JMX code in a few places > While the first may be sustainable, I'd expect that the 2nd option should be > put behind a configuration switch to allow the existing code to be used by > default and turning off JMX if configured. > *I haven't created a PR for now, as I'd like feedback to decide if it is > going to be feasible to carry this forwards.* -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (KAFKA-8629) Kafka Streams Apps to support small native images through GraalVM
[ https://issues.apache.org/jira/browse/KAFKA-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883067#comment-16883067 ] Staffan Olsson edited comment on KAFKA-8629 at 7/11/19 3:08 PM: Quarkus has a [Streams extension|[https://quarkus.io/guides/kafka-streams-guide]|https://quarkus.io/guides/kafka-streams-guide]. For details see [https://github.com/quarkusio/quarkus/pull/2693], [https://github.com/quarkusio/quarkus/issues/2863] and [https://github.com/quarkusio/quarkus/pull/2953.] Note also that IMO there's a blocker with Snappy and LZ4 disabled: [https://github.com/quarkusio/quarkus/issues/2718]. I've done some work on native builds of topic admin CLI tools, where the rewards in terms of reduced startup times really pay off: [https://github.com/solsson/dockerfiles/pull/25] ([example usage|https://github.com/Yolean/kubernetes-kafka/commit/af63569c41b78195b85c0ff7f919df93b2ef6003]). I've only used these builds in testing, but seen no regressions yet. They reduce CPU consumption for automated (by shell script) admin tasks by 90% and memory consumption by 80%. I did this work before I discovered [https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md] which would have made things a lot easier :) was (Author: solsson): Quarkus has a [Streams extension|[https://quarkus.io/guides/kafka-streams-guide]|https://quarkus.io/guides/kafka-streams-guide],]. For details see [https://github.com/quarkusio/quarkus/pull/2693], [https://github.com/quarkusio/quarkus/issues/2863] and [https://github.com/quarkusio/quarkus/pull/2953.] Note also that IMO there's a blocker with Snappy and LZ4 disabled: [https://github.com/quarkusio/quarkus/issues/2718]. I've done some work on native builds of topic admin CLI tools, where the rewards in terms of reduced startup times really pay off: [https://github.com/solsson/dockerfiles/pull/25] ([example usage|https://github.com/Yolean/kubernetes-kafka/commit/af63569c41b78195b85c0ff7f919df93b2ef6003]). I've only used these builds in testing, but seen no regressions yet. They reduce CPU consumption for automated (by shell script) admin tasks by 90% and memory consumption by 80%. I did this work before I discovered [https://github.com/oracle/graal/blob/master/substratevm/CONFIGURE.md] which would have made things a lot easier :) > Kafka Streams Apps to support small native images through GraalVM > - > > Key: KAFKA-8629 > URL: https://issues.apache.org/jira/browse/KAFKA-8629 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 > Environment: OSX > Linux on Docker >Reporter: Andy Muir >Priority: Minor > > I'm investigating using [GraalVM|http://example.com/] to help with reducing > docker image size and required resources for a simple Kafka Streams > microservice. To this end, I'm looking at running a microservice which: > 1) consumes from a Kafka topic (XML) > 2) Transforms into JSON > 3) Produces to a new Kafka topic. > The Kafka Streams app running in the JVM works fine. > When I attempt to build it to a GraalVM native image (binary executable which > does not require the JVM, hence smaller image size and less resources), I > encountered a few > [incompatibilities|https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md] > with the source code in Kafka. > I've implemented a workaround for each of these in a fork (link to follow) to > help establish if it is feasible. I don't intend (at this stage) for the > changes to be applied to the broker - I'm only after Kafka Streams for now. > I'm not sure whether it'd be a good idea for the broker itself to run as a > native image! > There were 2 issues getting the native image with kafka streams: > 1) Some Reflection use cases using MethodHandle > 2) Anything JMX > To work around these issues, I have: > 1) Replaced use of MethodHandle with alternatives > 2) Commented out the JMX code in a few places > While the first may be sustainable, I'd expect that the 2nd option should be > put behind a configuration switch to allow the existing code to be used by > default and turning off JMX if configured. > *I haven't created a PR for now, as I'd like feedback to decide if it is > going to be feasible to carry this forwards.* -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (KAFKA-8629) Kafka Streams Apps to support small native images through GraalVM
[ https://issues.apache.org/jira/browse/KAFKA-8629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878376#comment-16878376 ] Andy Muir edited comment on KAFKA-8629 at 7/4/19 4:42 PM: -- Kafka fork with modified code: [https://github.com/muirandy/kafka/tree/graal-native-image-support] Shows whats needed to make Kafka Streams app work with GraalVM native image. Note: More changes may be required, and as stated in the description above, this code is not ready for a PR. NB: This is my original work and I license the work to the project under the project's open source license. was (Author: muirandy): Kafka fork with modified code: [https://github.com/muirandy/kafka/tree/graal-native-image-support] Shows whats needed to make Kafka Streams app work with GraalVM native image. Note: More changes may be required, and as stated in the description above, this code is not ready for a PR. > Kafka Streams Apps to support small native images through GraalVM > - > > Key: KAFKA-8629 > URL: https://issues.apache.org/jira/browse/KAFKA-8629 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.3.0 > Environment: OSX > Linux on Docker >Reporter: Andy Muir >Priority: Minor > > I'm investigating using [GraalVM|http://example.com/] to help with reducing > docker image size and required resources for a simple Kafka Streams > microservice. To this end, I'm looking at running a microservice which: > 1) consumes from a Kafka topic (XML) > 2) Transforms into JSON > 3) Produces to a new Kafka topic. > The Kafka Streams app running in the JVM works fine. > When I attempt to build it to a GraalVM native image (binary executable which > does not require the JVM, hence smaller image size and less resources), I > encountered a few > [incompatibilities|https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md] > with the source code in Kafka. > I've implemented a workaround for each of these in a fork (link to follow) to > help establish if it is feasible. I don't intend (at this stage) for the > changes to be applied to the broker - I'm only after Kafka Streams for now. > I'm not sure whether it'd be a good idea for the broker itself to run as a > native image! > There were 2 issues getting the native image with kafka streams: > 1) Some Reflection use cases using MethodHandle > 2) Anything JMX > To work around these issues, I have: > 1) Replaced use of MethodHandle with alternatives > 2) Commented out the JMX code in a few places > While the first may be sustainable, I'd expect that the 2nd option should be > put behind a configuration switch to allow the existing code to be used by > default and turning off JMX if configured. > *I haven't created a PR for now, as I'd like feedback to decide if it is > going to be feasible to carry this forwards.* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
