I've updated the GH to reflect the idea outlined above Mike https://github.com/timrobertson100/kudu-test-server
- code is fairly hacky - deletes extracted binaries (FYI: deleteOnExit() will not delete dirs unless empty) - uses maven classifier (only linux exists) Can you PTAL Mike when you get a chance? - no rush at all On Thu, Aug 9, 2018 at 6:07 PM Tim <[email protected]> wrote: > Thanks Mike - no apologies needed at all > > I’ll aim to reshape the GH repo I did to illustrate what we outlined, with > an example of use. > > Outstanding is some OS X binaries if anyone has some time? > > Have a good trip. > > Tim, > Sent from my iPhone > > > On 9 Aug 2018, at 14:05, Mike Percy <[email protected]> wrote: > > > > Hi Tim, > > Sorry for the delay in responding, I've been trying to get back to this. > > You make some great points. If we can forego Maven/Gradle plugins, we can > > do this with a lot less work. Initially, I was concerned about unpacking > > the bits potentially more than strictly necessary, but it seems likely > that > > the CPU / IO required to do the unpacking would probably not be > noticeable > > in the grand scheme of running tests against a Kudu MiniCluster, > especially > > if it's only done once per test file (using @BeforeClass or similar). > Also, > > as long as there is some way to clean up the files that were unpacked and > > avoid potentially filling up a temp dir then nobody should have a problem > > with this approach... perhaps we can register a JVM shutdown hook and > also > > provide some kind of teardown() method so people can ensure the files get > > cleaned up as appropriate (I saw the TODO in your test code for this; > > protoc uses File.deleteOnExit()). > > > > Regarding creating the binary artifacts, I don't think it's too big of a > > burden to ask the release manager to upload either a RHEL 6 or macOS > binary > > test artifact based on the output of a script, and ask the PMC for help > to > > get a binary for the other platform. > > > > I'm out of town for the next couple of weeks but once I get back I'll see > > if I can push this forward a bit more. > > > > Thanks! > > Mike > > > > > > On Thu, Aug 2, 2018 at 10:57 PM Tim Robertson <[email protected] > > > > wrote: > > > >> Thanks to you for making this test possible Mike. > >> > >> I was approaching this emulating protoc where they put the binaries as > >> artifacts for Maven [1]. > >> A slight difference is that protoc is a single file while your tarball > has > >> several. I notice that the protoc build plugin for maven also copies out > >> from the jar to a local filesystem [2] so I just copied that approach. > >> > >> What I could imagine is we have a jar with the binaries and a single > class > >> (say EmbeddedKudu) that copies the binaries into a temporary directory > (as > >> my hack in GH). > >> > >> A user would then do the following: > >> > >> <!-- finds the environment --> > >> <build> > >> <extensions> > >> <extension> > >> <groupId>kr.motd.maven</groupId> > >> <artifactId>os-maven-plugin</artifactId> > >> <version>1.6.0</version> > >> </extension> > >> </extensions> > >> </build> > >> .... > >> <!-- Adds mini cluster --> > >> <dependency> > >> <groupId>org.apache.kudu</groupId> > >> <artifactId>kudu-client</artifactId> > >> <version>1.7.0</version> > >> <classifier>tests</classifier> > >> </dependency> > >> <!-- Adds an embedded Kudu --> > >> <dependency> > >> <groupId>org.apache.kudu</groupId> > >> <artifactId>kudu-test-server</artifactId> > >> <version>1.7.0</version> > >> <classifier>${os.detected.classifier}</classifier> > >> </dependency> > >> > >> > >> The detect classifier stuff is copied from protoc, but perhaps we'd just > >> state that available options are (?) linux / OSX and ignore > autodetection. > >> > >> And in code users would do something like: > >> > >> EmbeddedKudu.prepare(); // copies kudu from the jar and sets the > >> system variable (as per my example) > >> MiniKuduCluster miniCluster = > >> new > >> > MiniKuduCluster.MiniKuduClusterBuilder().numMasters(1).numTservers(1).build(); > >> > >> > >> What this would mean: > >> > >> - no change to the existing Kudu code packkaging > >> - leverage caching of the binaries as any Java artifact > >> - download would be at build time not test runtime - only copying out > from > >> the jar each time > >> - simple to include in most build environments (not tied to maven as a > >> plugin) > >> > >> I can't comment if it makes sense to do this WRT the binaries though > and it > >> would mean building and releasing binaries into a jar on each Kudu > release > >> (as protoc does). > >> > >> WDYT? > >> > >> Thanks, > >> Tim > >> > >> [1] http://repo1.maven.org/maven2/com/google/protobuf/protoc/3.6.1/ > >> [2] > >> > >> > https://github.com/os72/protoc-jar-maven-plugin/blob/master/src/main/java/com/github/os72/protocjar/maven/ProtocJarMojo.java#L664 > >> > >>> On Thu, Aug 2, 2018 at 10:28 PM, Mike Percy <[email protected]> wrote: > >>> > >>> Ha, neat, thanks for posting this, Tim. It's a nice proof of concept. > >>> > >>> I was imagining that we would try to implement the downloading part as > a > >>> Maven plugin, but maybe it could work to try to download the artifacts > at > >>> runtime with a JUnit test. Do you think we could cache the artifacts > >>> somewhere, maybe in the Maven repo somehow, so we don't have to > download > >>> the artifact every time we want to use it? I was hoping to simply be > able > >>> to ship a tarball or a jar of the binaries separately from the test > >>> framework code. > >>> > >>> Mike > >>> > >>> On Thu, Aug 2, 2018 at 8:42 AM Tim Robertson < > [email protected]> > >>> wrote: > >>> > >>>> Hi folks > >>>> > >>>> I've not had too much time, but I threw this together using Mike's > >>>> binaries: > >>>> https://github.com/timrobertson100/kudu-test-server > >>>> > >>>> I think this shows that running a mini cluster is possible using the > >>>> binaries Mike prepared when they are included in a jar (on CentOS 7.4 > >> at > >>>> least). > >>>> > >>>> Please don't flame me for the code - it was a rush job - but perhaps > >> you > >>>> could verify it works for you Mike? > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Thu, Jul 19, 2018 at 12:35 AM, Mike Percy <[email protected]> > >> wrote: > >>>> > >>>>> On Wed, Jul 18, 2018 at 1:24 PM Tim Robertson < > >>> [email protected] > >>>>> > >>>>> wrote: > >>>>> > >>>>>>> I'm pretty busy this week and would be happy to get some help on > >>> this > >>>>> if > >>>>>> anybody has cycles to spare. > >>>>>> > >>>>>> Thanks Mike - I'll look into 3) and if we get it working I'm happy > >> to > >>>>> offer > >>>>>> a PR for 4). > >>>>>> This is an evening project for me so I might be a little slow too - > >>> if > >>>>>> someone else is keen and has time please feel free to jump in. > >>>>>> > >>>>> > >>>>> Sounds great! I think Grant started working on #4 but I don't know > >> how > >>>> far > >>>>> he got. > >>>>> > >>>>> Mike > >>>>> > >>>> > >>> > >> >
