Nice work, Tim! Thanks for the update! Mike
On Tue, Jul 31, 2018 at 12:49 PM Tim Robertson <[email protected]> wrote: > Hey folks, > > Happy to let you know that Apache Beam master now has KuduIO (without > authentication) so that will be released with Beam 2.7.0. We're cutting > 2.6.0 now, and Beam has a 6 week release cycle. > > I tried Mike's binaries embedded in a Jar today but have not yet had > success. I'll continue to explore that as time allows but it is a side > project. > > Thanks, > Tim > > On Tue, Jul 24, 2018 at 7:49 PM, Mike Percy <[email protected]> wrote: > > > Hey Tim, very cool! I'll take a look at that patch and I'll get the WIP > > test artifact build script moved from gist to Gerrit in the next couple > of > > days. > > > > In the meantime, I just posted a link to a WIP binary test artifact for > > Linux on KUDU-2411. > > > > Mike > > > > On Tue, Jul 24, 2018 at 9:07 AM Tim Robertson <[email protected] > > > > wrote: > > > > > Hi folks > > > > > > Just to keep you updated - I offered a PR for first version(!) Kudu > > support > > > in Beam > > > https://github.com/apache/beam/pull/6021 > > > > > > Note: Authentication missing in the first implementation, but will be > > > added. > > > > > > You will see I have IT with instructions on using Docker, and have a > mix > > of > > > mock/fake-service for very basic unit test. This will be removed if we > > > crack the binary for *nix build as a maven artifact. > > > > > > This is very much a first implementation upon which we can improve. > > > > > > All comments or concerns very welcome, > > > Tim > > > > > > > > > On Tue, Jun 5, 2018 at 2:31 PM, Tim <[email protected]> wrote: > > > > > > > Thanks Attila > > > > > > > > That’s what I had expected. > > > > > > > > For info: Docker is required here because of all the build/test > > > > infrastructure within the Beam project - plus it needs to be portable > > > > across environments without devs needing to intervene for all the > > target > > > > systems. Binaries would presumably differ per environment right? > > > > > > > > I have Kudu running in Docker ok. > > > > > > > > Thank you also for offering to review - I appreciate it greatly. > > > > > > > > Tim, > > > > Sent from my iPhone > > > > > > > > > On 5 Jun 2018, at 13:50, Attila Bukor <[email protected]> wrote: > > > > > > > > > > Hi TIm, > > > > > > > > > > If you’re referring to MiniKuduCluster, it requires the Kudu > binaries > > > to > > > > start a MiniCluster which in turn uses the actual kudu-master and > > > > kudu-tserver binaries to set up a cluster, so it requires the Kudu > > > binaries > > > > to be available and will use separate processes, but you don’t have > to > > > set > > > > up an actual cluster beforehand. > > > > > > > > > > I agree that in unit tests Kudu should be mocked, but I don’t think > > > > Docker is necessary for the integration tests though, the MiniCluster > > > > should be used imho. > > > > > > > > > > I’d be happy to look at the pull request, but I’m not a committer, > so > > > it > > > > would be nice to have someone else look at it as well. > > > > > > > > > > Attila > > > > > > > > > >> On 2018. Jun 5., at 12:11, Tim Robertson < > [email protected] > > > > > > > wrote: > > > > >> > > > > >> Hi folks, > > > > >> > > > > >> I'm starting up development of a Java based connector to Kudu for > > > Apache > > > > >> Beam - i.e. a source and sink to Kudu for beam pipelines. > > > > >> > > > > >> For testing, the Beam project favours "mini integration tests" > > rather > > > > than > > > > >> mocking - e.g. spinning up an embedded HBase / Solr server. In > > > addition > > > > the > > > > >> beam project also has a set of separate integration tests that > make > > us > > > > use > > > > >> kubernetes. > > > > >> > > > > >> My understanding is for Kudu the embedded Java server is a wrapper > > > > around a > > > > >> running kudu cluster. I don't believe there is an embeddable Kudu > > for > > > > >> testing right (even an API compliant / mock server)? > > > > >> > > > > >> My current thinking is to mock Kudu in unit tests and provide an > > > > >> integration test using kudu running in Docker. Can anyone propose > a > > > > better > > > > >> option or confirm this sounds reasonable please? > > > > >> > > > > >> Once written would a kind Kudu dev be willing to take a look > before > > I > > > > put > > > > >> it up for a pull request in Beam perhaps? It would mean a lot to > > get a > > > > nod > > > > >> from you folk that it looks correct. > > > > >> > > > > >> Thanks, > > > > >> Tim > > > > > > > > > > > > > > > > > > > >
