[jira] [Commented] (KAFKA-10380) Make dist flatten rocksdbjni
[ https://issues.apache.org/jira/browse/KAFKA-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17181366#comment-17181366 ] Ewen Cheslack-Postava commented on KAFKA-10380: --- Probably the best solution would be to reorganize the binary dist to have subdirectories for clients, core, streams, connect, etc, but that get's tricky to because then bin scripts need to deal with more classpath stuff or jars get duplicated, we'd need to figure out how config files get organized, etc. But that would substantially reduce certain use cases, e.g. connect also pulls in jetty/jersey since it was a REST API, both of which are pretty large with their transitive dependencies. > Make dist flatten rocksdbjni > > > Key: KAFKA-10380 > URL: https://issues.apache.org/jira/browse/KAFKA-10380 > Project: Kafka > Issue Type: Task > Components: build >Affects Versions: 2.6.0 >Reporter: Adrian Cole >Priority: Major > > I was looking for ways to reduce the size of our Kafka image, and the most > notable opportunity is handling rocksdbjni differently. It is currently a > 15MB jar. > As mentioned in its description rocksdbjni includes binaries for a lot of OS > choices. > du -k librocksdbjni-* > 7220 librocksdbjni-linux-aarch64.so > 8756 librocksdbjni-linux-ppc64le.so > 7220 librocksdbjni-linux32.so > 7932 librocksdbjni-linux64.so > 5440 librocksdbjni-osx.jnilib > 4616 librocksdbjni-win64.dll > It may not seem obvious in normal dists, which aim to work for many operating > systems what is a problem here. When creating docker images, we currently > would need to repackage this to scrub out the irrelevant OS items or accept > files larger than alpine itself. > While this might be something to kick back to rocksdb. having some options > here would be great. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10380) Make dist flatten rocksdbjni
[ https://issues.apache.org/jira/browse/KAFKA-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17180889#comment-17180889 ] Matthias J. Sax commented on KAFKA-10380: - Thanks for clarifying. I am sure that there are other libraries in the binary dist that are not required for the brokers and I understand that for a "minimal" broker image it's not easy to know which one you need and which ones could be omitted. – Not sure atm how to address this in the most straightforward way. \cc [~ijuma] and or [~ewencp] might have an opinion about this question. > Make dist flatten rocksdbjni > > > Key: KAFKA-10380 > URL: https://issues.apache.org/jira/browse/KAFKA-10380 > Project: Kafka > Issue Type: Task > Components: build >Affects Versions: 2.6.0 >Reporter: Adrian Cole >Priority: Major > > I was looking for ways to reduce the size of our Kafka image, and the most > notable opportunity is handling rocksdbjni differently. It is currently a > 15MB jar. > As mentioned in its description rocksdbjni includes binaries for a lot of OS > choices. > du -k librocksdbjni-* > 7220 librocksdbjni-linux-aarch64.so > 8756 librocksdbjni-linux-ppc64le.so > 7220 librocksdbjni-linux32.so > 7932 librocksdbjni-linux64.so > 5440 librocksdbjni-osx.jnilib > 4616 librocksdbjni-win64.dll > It may not seem obvious in normal dists, which aim to work for many operating > systems what is a problem here. When creating docker images, we currently > would need to repackage this to scrub out the irrelevant OS items or accept > files larger than alpine itself. > While this might be something to kick back to rocksdb. having some options > here would be great. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10380) Make dist flatten rocksdbjni
[ https://issues.apache.org/jira/browse/KAFKA-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17180886#comment-17180886 ] Adrian Cole commented on KAFKA-10380: - I'm referring to the binary dist (produced by the build). The problem is that this lib is in mixed with the others so it is hard to know which jars are optional for brokers or connect. I can make a special comment in our file about this one, but maybe a documentation or directory naming convention could be used to identify what's not required for basic kafka functionality (vs streams). since this seems a special case, it is ok to just add a comment, just mentioning there's no way for a passerby to know this is only needed for streams. there may be other larger jars also unused at runtime, so if you know others, let me know also! > Make dist flatten rocksdbjni > > > Key: KAFKA-10380 > URL: https://issues.apache.org/jira/browse/KAFKA-10380 > Project: Kafka > Issue Type: Task > Components: build >Affects Versions: 2.6.0 >Reporter: Adrian Cole >Priority: Major > > I was looking for ways to reduce the size of our Kafka image, and the most > notable opportunity is handling rocksdbjni differently. It is currently a > 15MB jar. > As mentioned in its description rocksdbjni includes binaries for a lot of OS > choices. > du -k librocksdbjni-* > 7220 librocksdbjni-linux-aarch64.so > 8756 librocksdbjni-linux-ppc64le.so > 7220 librocksdbjni-linux32.so > 7932 librocksdbjni-linux64.so > 5440 librocksdbjni-osx.jnilib > 4616 librocksdbjni-win64.dll > It may not seem obvious in normal dists, which aim to work for many operating > systems what is a problem here. When creating docker images, we currently > would need to repackage this to scrub out the irrelevant OS items or accept > files larger than alpine itself. > While this might be something to kick back to rocksdb. having some options > here would be great. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-10380) Make dist flatten rocksdbjni
[ https://issues.apache.org/jira/browse/KAFKA-10380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17180878#comment-17180878 ] Matthias J. Sax commented on KAFKA-10380: - [~adriancole] – when you say "Kafka image" what do you exactly mean? RocksDB should only be a dependency for Kafka Streams, but not for brokers or connect. > Make dist flatten rocksdbjni > > > Key: KAFKA-10380 > URL: https://issues.apache.org/jira/browse/KAFKA-10380 > Project: Kafka > Issue Type: Task > Components: build >Affects Versions: 2.6.0 >Reporter: Adrian Cole >Priority: Major > > I was looking for ways to reduce the size of our Kafka image, and the most > notable opportunity is handling rocksdbjni differently. It is currently a > 15MB jar. > As mentioned in its description rocksdbjni includes binaries for a lot of OS > choices. > du -k librocksdbjni-* > 7220 librocksdbjni-linux-aarch64.so > 8756 librocksdbjni-linux-ppc64le.so > 7220 librocksdbjni-linux32.so > 7932 librocksdbjni-linux64.so > 5440 librocksdbjni-osx.jnilib > 4616 librocksdbjni-win64.dll > It may not seem obvious in normal dists, which aim to work for many operating > systems what is a problem here. When creating docker images, we currently > would need to repackage this to scrub out the irrelevant OS items or accept > files larger than alpine itself. > While this might be something to kick back to rocksdb. having some options > here would be great. -- This message was sent by Atlassian Jira (v8.3.4#803005)
