Hi Tim, Sorry for the delay in responding, I've been trying to get back to this. You make some great points. If we can forego Maven/Gradle plugins, we can do this with a lot less work. Initially, I was concerned about unpacking the bits potentially more than strictly necessary, but it seems likely that the CPU / IO required to do the unpacking would probably not be noticeable in the grand scheme of running tests against a Kudu MiniCluster, especially if it's only done once per test file (using @BeforeClass or similar). Also, as long as there is some way to clean up the files that were unpacked and avoid potentially filling up a temp dir then nobody should have a problem with this approach... perhaps we can register a JVM shutdown hook and also provide some kind of teardown() method so people can ensure the files get cleaned up as appropriate (I saw the TODO in your test code for this; protoc uses File.deleteOnExit()).
Regarding creating the binary artifacts, I don't think it's too big of a burden to ask the release manager to upload either a RHEL 6 or macOS binary test artifact based on the output of a script, and ask the PMC for help to get a binary for the other platform. I'm out of town for the next couple of weeks but once I get back I'll see if I can push this forward a bit more. Thanks! Mike On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <[email protected]> wrote: > Thanks to you for making this test possible Mike. > > I was approaching this emulating protoc where they put the binaries as > artifacts for Maven [1]. > A slight difference is that protoc is a single file while your tarball has > several. I notice that the protoc build plugin for maven also copies out > from the jar to a local filesystem [2] so I just copied that approach. > > What I could imagine is we have a jar with the binaries and a single class > (say EmbeddedKudu) that copies the binaries into a temporary directory (as > my hack in GH). > > A user would then do the following: > > <!-- finds the environment --> > <build> > <extensions> > <extension> > <groupId>kr.motd.maven</groupId> > <artifactId>os-maven-plugin</artifactId> > <version>1.6.0</version> > </extension> > </extensions> > </build> > .... > <!-- Adds mini cluster --> > <dependency> > <groupId>org.apache.kudu</groupId> > <artifactId>kudu-client</artifactId> > <version>1.7.0</version> > <classifier>tests</classifier> > </dependency> > <!-- Adds an embedded Kudu --> > <dependency> > <groupId>org.apache.kudu</groupId> > <artifactId>kudu-test-server</artifactId> > <version>1.7.0</version> > <classifier>${os.detected.classifier}</classifier> > </dependency> > > > The detect classifier stuff is copied from protoc, but perhaps we'd just > state that available options are (?) linux / OSX and ignore autodetection. > > And in code users would do something like: > > EmbeddedKudu.prepare(); // copies kudu from the jar and sets the > system variable (as per my example) > MiniKuduCluster miniCluster = > new > MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build(); > > > What this would mean: > > - no change to the existing Kudu code packkaging > - leverage caching of the binaries as any Java artifact > - download would be at build time not test runtime - only copying out from > the jar each time > - simple to include in most build environments (not tied to maven as a > plugin) > > I can't comment if it makes sense to do this WRT the binaries though and it > would mean building and releasing binaries into a jar on each Kudu release > (as protoc does). > > WDYT? > > Thanks, > Tim > > [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/ > [2] > > https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664 > > On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <[email protected]> wrote: > > > Ha, neat, thanks for posting this, Tim. It's a nice proof of concept. > > > > I was imagining that we would try to implement the downloading part as a > > Maven plugin, but maybe it could work to try to download the artifacts at > > runtime with a JUnit test. Do you think we could cache the artifacts > > somewhere, maybe in the Maven repo somehow, so we don't have to download > > the artifact every time we want to use it? I was hoping to simply be able > > to ship a tarball or a jar of the binaries separately from the test > > framework code. > > > > Mike > > > > On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson <[email protected]> > > wrote: > > > > > Hi folks > > > > > > I've not had too much time, but I threw this together using Mike's > > > binaries: > > > https://github.com/timrobertson100/kudu-test-server > > > > > > I think this shows that running a mini cluster is possible using the > > > binaries Mike prepared when they are included in a jar (on CentOS 7.4 > at > > > least). > > > > > > Please don't flame me for the code - it was a rush job - but perhaps > you > > > could verify it works for you Mike? > > > > > > > > > > > > > > > > > > > > > On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <[email protected]> > wrote: > > > > > > > On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson < > > [email protected] > > > > > > > > wrote: > > > > > > > > > > I'm pretty busy this week and would be happy to get some help on > > this > > > > if > > > > > anybody has cycles to spare. > > > > > > > > > > Thanks Mike - I'll look into 3) and if we get it working I'm happy > to > > > > offer > > > > > a PR for 4). > > > > > This is an evening project for me so I might be a little slow too - > > if > > > > > someone else is keen and has time please feel free to jump in. > > > > > > > > > > > > > Sounds great! I think Grant started working on #4 but I don't know > how > > > far > > > > he got. > > > > > > > > Mike > > > > > > > > > >
