Hi folks Just to keep you updated - I offered a PR for first version(!) Kudu support in Beam https://github.com/apache/beam/pull/6021
Note: Authentication missing in the first implementation, but will be added. You will see I have IT with instructions on using Docker, and have a mix of mock/fake-service for very basic unit test. This will be removed if we crack the binary for *nix build as a maven artifact. This is very much a first implementation upon which we can improve. All comments or concerns very welcome, Tim On Tue, Jun 5, 2018 at 2:31 PM, Tim <[email protected]> wrote: > Thanks Attila > > That’s what I had expected. > > For info: Docker is required here because of all the build/test > infrastructure within the Beam project - plus it needs to be portable > across environments without devs needing to intervene for all the target > systems. Binaries would presumably differ per environment right? > > I have Kudu running in Docker ok. > > Thank you also for offering to review - I appreciate it greatly. > > Tim, > Sent from my iPhone > > > On 5 Jun 2018, at 13:50, Attila Bukor <[email protected]> wrote: > > > > Hi TIm, > > > > If you’re referring to MiniKuduCluster, it requires the Kudu binaries to > start a MiniCluster which in turn uses the actual kudu-master and > kudu-tserver binaries to set up a cluster, so it requires the Kudu binaries > to be available and will use separate processes, but you don’t have to set > up an actual cluster beforehand. > > > > I agree that in unit tests Kudu should be mocked, but I don’t think > Docker is necessary for the integration tests though, the MiniCluster > should be used imho. > > > > I’d be happy to look at the pull request, but I’m not a committer, so it > would be nice to have someone else look at it as well. > > > > Attila > > > >> On 2018. Jun 5., at 12:11, Tim Robertson <[email protected]> > wrote: > >> > >> Hi folks, > >> > >> I'm starting up development of a Java based connector to Kudu for Apache > >> Beam - i.e. a source and sink to Kudu for beam pipelines. > >> > >> For testing, the Beam project favours "mini integration tests" rather > than > >> mocking - e.g. spinning up an embedded HBase / Solr server. In addition > the > >> beam project also has a set of separate integration tests that make us > use > >> kubernetes. > >> > >> My understanding is for Kudu the embedded Java server is a wrapper > around a > >> running kudu cluster. I don't believe there is an embeddable Kudu for > >> testing right (even an API compliant / mock server)? > >> > >> My current thinking is to mock Kudu in unit tests and provide an > >> integration test using kudu running in Docker. Can anyone propose a > better > >> option or confirm this sounds reasonable please? > >> > >> Once written would a kind Kudu dev be willing to take a look before I > put > >> it up for a pull request in Beam perhaps? It would mean a lot to get a > nod > >> from you folk that it looks correct. > >> > >> Thanks, > >> Tim > > > > >
