On Mon, Jul 2, 2018 at 12:24 PM, Mike Percy <mpe...@apache.org> wrote:
> > So it's not viable to simply have a linux-x86_64 binary and a darwin-x86_64 > binary like protoc does, or even just ubuntu & redhat. We'll likely need a > separate binary for every major OS version, i.e. RHEL 6, RHEL 7, trusty, > xenial, bionic. I think people running non-LTS builds of Ubuntu, or SUSE or > something, would be out of luck. > In many cases, builds done on earlier systems are runnable on newer systems. For example, el6 builds tend to run fine on el7 and ubuntu 14 in my experience. If you also bundle the libssl and libcrypto, it seems that an el6 build can also run on Ubuntu 16. So, assuming we could package more than one binary in the artifact, packaging the shared libraries and setting RPATH or LD_LIBRARY_PATH accordingly is probably a reasonable option to run across most common Linux variants. -Todd > > On Sat, Jun 30, 2018 at 4:31 AM Tim Robertson <timrobertson...@gmail.com> > wrote: > > > > What do you mean by that? > > Sorry, poor phrasing - currently the Beam project has the build path with > > unit tests (no Docker there) and the project IT environment which can use > > Docker. > > A binary only approach could potentially be managed without adding a > > dependency on Docker - but has other issues summarised below. > > > > > For Kudu-internal testing I think we could stick to running "kudu > > minicluster > > Yes. > > > > > ... external use cases, we could switch that to "docker run > > kudu:minicluster:1.7.0" > > I think this makes good sense. > > > > > > In summary: > > > > 1) Fake a Kudu master in Java - difficult unless simplified, not > > representative if simplified, code maintenance issue > > 2) Mocking the Kudu client - verbose unless only covering simple > scenarios > > 3) Use mini cluster with binaries - portability challenge of binaries, > need > > to script caching the binaries / use of some repository, unfamiliar build > > tasks with binary handling (unless built to work with something like > > maven), possible could see linking problems > > 4) Docker - predictable, adds a dependency, existing Kudu images not > > "managed" at the moment > > > > For Beam I think I will put most effort into IT which can use Docker or > an > > existing cluster and then mock a Java KuduClient for some basic sanity > > tests for the build path. > > > > On Docker: > > - to get current versions [e.g. 1] working I found I had to edit > > /etc/hosts. I think the mini cluster version with the FakeDNS might avoid > > that? > > - Kudu docs currently encourage the Cloudera Quickstart VM over Docker > > [2,3] > > > > Do you think the Kudu project could provide an image allowing "docker run > > kudu:minicluster:1.x.x" as part of the release cycle? > > > > Thanks again, > > Tim > > > > [1] https://github.com/MartinWeindel/kudu-docker > > [2] https://kudu.apache.org/docs/quickstart.html#quickstart_vm > > [3] https://github.com/cloudera/kudu-examples/wiki/Docker-based-tutorial > > > > On Sat, Jun 30, 2018 at 2:22 AM, Todd Lipcon <t...@cloudera.com.invalid> > > wrote: > > > > > On Fri, Jun 29, 2018 at 1:23 PM, Tim Robertson < > > timrobertson...@gmail.com> > > > wrote: > > > > > > > Thanks Mike, Todd - I greatly appreciate the inputs. > > > > > > > > > How many platforms would need to be supported for it to be viable > for > > > > Beam? > > > > The minimal for it to be considered would probably(!) be ubuntu, > > centos, > > > > osx. Incidentally it was actually the protobuf approach that make me > > > > consider this. > > > > > > > > > What about depending on a docker container than runs the kudu > > > > minicluster in > > > > "host" networking mode? > > > > I've also pondered this a little but like Attila raises it puts a lot > > of > > > > burden for other project developers. Mmmm... > > > > > > > > > > What do you mean by that? For Kudu-internal testing I think we could > > stick > > > to running "kudu minicluster" as is. For external use cases, we could > > > switch that to "docker run kudu:minicluster:1.7.0" or whatever, and it > > > would auto-download from dockerhub as necessary, right? > > > > > > > > > > > > > > Ismaël (Beam PMC) has suggested I stick to mocking given the > complexity > > > of > > > > the things I'm exploring. > > > > > > > > As another idea: > > > > I briefly pondered writing a "FakeKudu Java server" - data held in > > > memory, > > > > no partitioning, protobuf messaging, handling table metadata, > checking > > > > schemas on write, predicate and projected columns for scan, faking > > > kerberos > > > > (if possible). It didn't seem particularly difficult to do but I > fear a > > > > maintenance burden for a small audience. > > > > > > > > > > > Yea, I think that would be quite a maintenance burden, especially as > new > > > features are added over time. I suppose in many cases you could omit > > things > > > or stub things out, but then the behavior will begin to differ and it > > won't > > > really be that clear that your tests actually are representative. > > > > > > > > > > Could utilities in Kudu that help folk test Java clients be of > interest > > > to > > > > others? - e.g. preconfigured mock objects for various scenarios. If > so, > > > I'd > > > > be happy to discuss options and offer PRs in Kudu. > > > > > > > > Thanks, > > > > Tim > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jun 29, 2018 at 9:34 PM, Todd Lipcon > <t...@cloudera.com.invalid > > > > > > > wrote: > > > > > > > > > On Fri, Jun 29, 2018 at 12:31 PM, Mike Percy <mpe...@apache.org> > > > wrote: > > > > > > > > > > > This is something I've been thinking about and toying with and > I'd > > > like > > > > > to > > > > > > see if we can't get binaries available via Maven for at least one > > > > > platform > > > > > > (say, RHEL 7). Similar to how protobuf does it. > > > > > > > > > > > > > > > > What about depending on a docker container than runs the kudu > > > minicluster > > > > > in "host" networking mode? eg https://github.com/ > > > > MartinWeindel/kudu-docker > > > > > is one possibility > > > > > > > > > > > > > > > > How many platforms would need to be supported for it to be viable > > for > > > > > Beam? > > > > > > > > > > > > Thanks, > > > > > > Mike > > > > > > > > > > > > On Fri, Jun 29, 2018 at 10:01 AM Tim <timrobertson...@gmail.com> > > > > wrote: > > > > > > > > > > > > > Thanks Attila > > > > > > > > > > > > > > That’s great feedback and helpful for me to reference as > > guidance. > > > > > > > > > > > > > > By “Kudu installation” I was referring to the possibility that > an > > > > > install > > > > > > > might set config etc, beyond just having the binary. I got it > > > running > > > > > on > > > > > > > CentOS similar to how you outline now. > > > > > > > > > > > > > > I too believe mocking makes most sense, especially as we have > the > > > IT > > > > > > > running as well, but was asked to explore this further. It’s > > useful > > > > to > > > > > > know > > > > > > > you’d agree. > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > Tim > > > > > > > > > > > > > > > On 29 Jun 2018, at 17:33, Attila Bukor <abu...@cloudera.com> > > > > wrote: > > > > > > > > > > > > > > > > Hi Tim, > > > > > > > > > > > > > > > > I’m not sure what you mean by relying on actual > installations. > > If > > > > you > > > > > > > have the kudu, kudu-master and kudu-tserver binaries at the > same > > > > > location > > > > > > > and they can be executed, MiniKuduCluster can be used (“binDir” > > > > > property > > > > > > > should be set to the directory containing the Kudu binaries). > You > > > > > should > > > > > > > also look into BaseKuduTest as that will set up the > > MiniKuduCluster > > > > for > > > > > > you > > > > > > > and you don’t have to do it manually. > > > > > > > > > > > > > > > > Extracting the Kudu binaries from an rpm should probably > work, > > > but > > > > > that > > > > > > > binds you to CDH as currently Cloudera is the only one that > ships > > > > Kudu > > > > > > > binaries and MacOS builds are not available anywhere afaik. > Also, > > > > 1.4.0 > > > > > > is > > > > > > > around a year old, you might want to use this repository > instead > > > > (from > > > > > > CDH > > > > > > > 5.13 Kudu is part of the CDH): > > > > > > > http://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5/ > > > > > > RPMS/x86_64/kudu-1.7.0+cdh5.15.0+0-1.cdh5.15.0.p0.52.el7. > x86_64.rpm > > > > > > > > > > > > > > > > As a general suggestion, I would recommend mocking Kudu for > > unit > > > > > tests > > > > > > > (that’s what a unit test is for after all) and create separate > > > > > > integration > > > > > > > tests that actually use Kudu that can be skipped where Kudu is > > not > > > > > > > available. Of course the CI should be set up to be able to > > provide > > > > all > > > > > > > necessary integrations for the tests, but a developer wouldn’t > > have > > > > to > > > > > > set > > > > > > > up Kudu, or use Docker to run the tests if their change doesn’t > > > > affect > > > > > > the > > > > > > > Kudu integration. > > > > > > > > > > > > > > > > Attila > > > > > > > > > > > > > > > >> On 2018. Jun 29., at 16:42, Tim Robertson < > > > > > timrobertson...@gmail.com> > > > > > > > wrote: > > > > > > > >> > > > > > > > >> Hi folks, > > > > > > > >> > > > > > > > >> I've written Java KuduIO for Apache Beam with integration > > tests > > > > > making > > > > > > > use > > > > > > > >> of Kudu in Docker. It is yet to be committed on Apache > Beam. > > > > > > > >> > > > > > > > >> Rather than mocking Kudu client for unit tests I'd like to > > > explore > > > > > use > > > > > > > of > > > > > > > >> the MiniKuduCluster which "Depends on precompiled kudu, > > > > kudu-master, > > > > > > and > > > > > > > >> kudu-tserver binaries". > > > > > > > >> > > > > > > > >> I'd need unit tests to run on the main linux distros and OS > X. > > > > > > > >> > > > > > > > >> For the linux distros, would an approach where I extract the > > > > > binaries > > > > > > > from > > > > > > > >> the packages [1] work please? Or does the MiniKuduCluster > rely > > > on > > > > > > actual > > > > > > > >> installations? I am pretty weak on C builds and linked > > libraries > > > > etc > > > > > > > (Java > > > > > > > >> guy, sorry). > > > > > > > >> > > > > > > > >> For CentOS I'm exploring this for example: > > > > > > > >> rpm2cpio ./kudu-1.4.0+cdh5.12.2+0-1. > > > > cdh5.12.2.p0.8.el7.x86_64.rpm > > > > > | > > > > > > > cpio > > > > > > > >> -idmv > > > > > > > >> > > > > > > > >> I haven't explored OS X options yet. > > > > > > > >> > > > > > > > >> Any advice here would greatly be appreciated to save me > going > > > > down a > > > > > > > dead > > > > > > > >> end. > > > > > > > >> > > > > > > > >> Many thanks, > > > > > > > >> Tim > > > > > > > >> > > > > > > > >> > > > > > > > >> [1] http://kudu.apache.org/docs/installation.html#install_ > > > > packages > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Todd Lipcon > > > > > Software Engineer, Cloudera > > > > > > > > > > > > > > > > > > > > > -- > > > Todd Lipcon > > > Software Engineer, Cloudera > > > > > > -- Todd Lipcon Software Engineer, Cloudera
