Hey Tim, very cool! I'll take a look at that patch and I'll get the WIP test artifact build script moved from gist to Gerrit in the next couple of days.
In the meantime, I just posted a link to a WIP binary test artifact for Linux on KUDU-2411. Mike On Tue, Jul 24, 2018 at 9:07 AM Tim Robertson <[email protected]> wrote: > Hi folks > > Just to keep you updated - I offered a PR for first version(!) Kudu support > in Beam > https://github.com/apache/beam/pull/6021 > > Note: Authentication missing in the first implementation, but will be > added. > > You will see I have IT with instructions on using Docker, and have a mix of > mock/fake-service for very basic unit test. This will be removed if we > crack the binary for *nix build as a maven artifact. > > This is very much a first implementation upon which we can improve. > > All comments or concerns very welcome, > Tim > > > On Tue, Jun 5, 2018 at 2:31 PM, Tim <[email protected]> wrote: > > > Thanks Attila > > > > That’s what I had expected. > > > > For info: Docker is required here because of all the build/test > > infrastructure within the Beam project - plus it needs to be portable > > across environments without devs needing to intervene for all the target > > systems. Binaries would presumably differ per environment right? > > > > I have Kudu running in Docker ok. > > > > Thank you also for offering to review - I appreciate it greatly. > > > > Tim, > > Sent from my iPhone > > > > > On 5 Jun 2018, at 13:50, Attila Bukor <[email protected]> wrote: > > > > > > Hi TIm, > > > > > > If you’re referring to MiniKuduCluster, it requires the Kudu binaries > to > > start a MiniCluster which in turn uses the actual kudu-master and > > kudu-tserver binaries to set up a cluster, so it requires the Kudu > binaries > > to be available and will use separate processes, but you don’t have to > set > > up an actual cluster beforehand. > > > > > > I agree that in unit tests Kudu should be mocked, but I don’t think > > Docker is necessary for the integration tests though, the MiniCluster > > should be used imho. > > > > > > I’d be happy to look at the pull request, but I’m not a committer, so > it > > would be nice to have someone else look at it as well. > > > > > > Attila > > > > > >> On 2018. Jun 5., at 12:11, Tim Robertson <[email protected]> > > wrote: > > >> > > >> Hi folks, > > >> > > >> I'm starting up development of a Java based connector to Kudu for > Apache > > >> Beam - i.e. a source and sink to Kudu for beam pipelines. > > >> > > >> For testing, the Beam project favours "mini integration tests" rather > > than > > >> mocking - e.g. spinning up an embedded HBase / Solr server. In > addition > > the > > >> beam project also has a set of separate integration tests that make us > > use > > >> kubernetes. > > >> > > >> My understanding is for Kudu the embedded Java server is a wrapper > > around a > > >> running kudu cluster. I don't believe there is an embeddable Kudu for > > >> testing right (even an API compliant / mock server)? > > >> > > >> My current thinking is to mock Kudu in unit tests and provide an > > >> integration test using kudu running in Docker. Can anyone propose a > > better > > >> option or confirm this sounds reasonable please? > > >> > > >> Once written would a kind Kudu dev be willing to take a look before I > > put > > >> it up for a pull request in Beam perhaps? It would mean a lot to get a > > nod > > >> from you folk that it looks correct. > > >> > > >> Thanks, > > >> Tim > > > > > > > > >
