Hey folks, Happy to let you know that Apache Beam master now has KuduIO (without authentication) so that will be released with Beam 2.7.0. We're cutting 2.6.0 now, and Beam has a 6 week release cycle.
I tried Mike's binaries embedded in a Jar today but have not yet had success. I'll continue to explore that as time allows but it is a side project. Thanks, Tim On Tue, Jul 24, 2018 at 7:49 PM, Mike Percy <[email protected]> wrote: > Hey Tim, very cool! I'll take a look at that patch and I'll get the WIP > test artifact build script moved from gist to Gerrit in the next couple of > days. > > In the meantime, I just posted a link to a WIP binary test artifact for > Linux on KUDU-2411. > > Mike > > On Tue, Jul 24, 2018 at 9:07 AM Tim Robertson <[email protected]> > wrote: > > > Hi folks > > > > Just to keep you updated - I offered a PR for first version(!) Kudu > support > > in Beam > > https://github.com/apache/beam/pull/6021 > > > > Note: Authentication missing in the first implementation, but will be > > added. > > > > You will see I have IT with instructions on using Docker, and have a mix > of > > mock/fake-service for very basic unit test. This will be removed if we > > crack the binary for *nix build as a maven artifact. > > > > This is very much a first implementation upon which we can improve. > > > > All comments or concerns very welcome, > > Tim > > > > > > On Tue, Jun 5, 2018 at 2:31 PM, Tim <[email protected]> wrote: > > > > > Thanks Attila > > > > > > That’s what I had expected. > > > > > > For info: Docker is required here because of all the build/test > > > infrastructure within the Beam project - plus it needs to be portable > > > across environments without devs needing to intervene for all the > target > > > systems. Binaries would presumably differ per environment right? > > > > > > I have Kudu running in Docker ok. > > > > > > Thank you also for offering to review - I appreciate it greatly. > > > > > > Tim, > > > Sent from my iPhone > > > > > > > On 5 Jun 2018, at 13:50, Attila Bukor <[email protected]> wrote: > > > > > > > > Hi TIm, > > > > > > > > If you’re referring to MiniKuduCluster, it requires the Kudu binaries > > to > > > start a MiniCluster which in turn uses the actual kudu-master and > > > kudu-tserver binaries to set up a cluster, so it requires the Kudu > > binaries > > > to be available and will use separate processes, but you don’t have to > > set > > > up an actual cluster beforehand. > > > > > > > > I agree that in unit tests Kudu should be mocked, but I don’t think > > > Docker is necessary for the integration tests though, the MiniCluster > > > should be used imho. > > > > > > > > I’d be happy to look at the pull request, but I’m not a committer, so > > it > > > would be nice to have someone else look at it as well. > > > > > > > > Attila > > > > > > > >> On 2018. Jun 5., at 12:11, Tim Robertson <[email protected] > > > > > wrote: > > > >> > > > >> Hi folks, > > > >> > > > >> I'm starting up development of a Java based connector to Kudu for > > Apache > > > >> Beam - i.e. a source and sink to Kudu for beam pipelines. > > > >> > > > >> For testing, the Beam project favours "mini integration tests" > rather > > > than > > > >> mocking - e.g. spinning up an embedded HBase / Solr server. In > > addition > > > the > > > >> beam project also has a set of separate integration tests that make > us > > > use > > > >> kubernetes. > > > >> > > > >> My understanding is for Kudu the embedded Java server is a wrapper > > > around a > > > >> running kudu cluster. I don't believe there is an embeddable Kudu > for > > > >> testing right (even an API compliant / mock server)? > > > >> > > > >> My current thinking is to mock Kudu in unit tests and provide an > > > >> integration test using kudu running in Docker. Can anyone propose a > > > better > > > >> option or confirm this sounds reasonable please? > > > >> > > > >> Once written would a kind Kudu dev be willing to take a look before > I > > > put > > > >> it up for a pull request in Beam perhaps? It would mean a lot to > get a > > > nod > > > >> from you folk that it looks correct. > > > >> > > > >> Thanks, > > > >> Tim > > > > > > > > > > > > > >
