Yes. Flume core could be considered to be the flume-ng-configuration, flume-ng-node, flume-ng-sdk, and most of flume-ng-core modules. While there are a few things in flume-ng-core worth pulling out I wouldn’t remove them initially unless they obviously fit into one of the repos I listed below. I should also add that the File Channel is a child module of the flume-ng-channels module. I personally also consider that to be a core component.
I should also add that the main flume build should add a flume-bom so that if the repos are versioned independently users would still have a way of including them easily. Ralph > On Oct 13, 2022, at 9:22 PM, Matt Sicker <boa...@gmail.com> wrote: > > Is there some sort of flume core, too, or is that the spring boot one? > > — > Matt Sicker > >> On Oct 13, 2022, at 23:03, Ralph Goers <ralph.go...@dslextreme.com> wrote: >> >> The Flume build takes forever. It takes about 70 minutes on my 2019 MacBook >> Pro, which is a pretty beefy machine. In addition, the build continuously >> fails in the CI builds because it generates too much output and runs out of >> disk space. >> >> I discussed this previously but I would like to start breaking up Flume into >> separate, independently releases repos. This should make releases easier. >> >> Another point of discussion would be that Flume is currently released as a >> packaged zip. Personally, I think this is a bad idea as it includes ALL the >> Flume components whether they are required or not. It makes more sense to me >> to build Flume as a normal application using Maven dependencies. If you use >> the new Spring Boot support you will still get all the dependencies packaged >> in the deployable jar. Even if you don’t like using Spring Boot I believe >> you can still use the Spring Boot Maven plugin to generate an executable jar. >> >> To start this discussion off I would propose immediately creating the >> following repos. >> >> 1. flume-spring-boot >> 2. flume-kafka >> 3. flume-jdbc >> 4. flume-legacy >> 5. flume-hadoop (would contain hive, hdfs, and hbase stuff) >> 6. flume-kudu >> 7. flume-jms >> 8. flume-twitter >> 9. flume-thrift. >> >> In addition, flume-search was already created and I would like to move the >> flume-ng-morphline-solr-sink there. For the time being the Elasticsearch >> module will need to be bypassed until it can be upgraded to a supportable >> version of ES. >> >> Thoughts? >> >> Ralph
