Kafka connector for Beam Python SDK

Hi All, I'm looking into adding a Kafka connector to Beam Python SDK. I think this will benefits many Python SDK users and will serve as a good example for recently added Splittable DoFn API (Fn API support which will allow all runners to use Python Splittable DoFn is in active development). I

Re: Apache Beam - jenkins question

Thanks for the heads-up regarding the permissions. At this point I need more information about the credentials we want to use -- Kamil, can you provide more info? What is the purpose of the credentials you want to use here? On Fri, Apr 27, 2018 at 3:50 PM Davor Bonaci wrote: >

Re: Apache Beam - jenkins question

Jason, you should now have all the permissions needed. (You should, however, evaluate whether this is a good place for it. Executors themselves, for example, might be an alternative.) On Fri, Apr 27, 2018 at 7:42 PM, Jason Kuster wrote: > See

Pubsub on directrunner: direct_runner.py and transform_evaluator.py

Hi, I'm having trouble understanding why there's an extra level of indirection when doing pubsub reads via directrunner vs writes. For reads, we have these translations: beam_pubsub.ReadFromPubSub -> direct_runner._DirectReadFromPubSub -> transform_evaluator._PubSubReadEvaluator For writes, this

Re: Which features of Dropwizard do you wish to use most with Beam metrics?

Thank you both very much, appreciate this feedback :) On Wed, Apr 25, 2018 at 9:47 PM Jean-Baptiste Onofré wrote: > Hi Alex > > I think counter and gauge is a good start. Timeline histograms are less > useful at first glance IMHO. > > Regards > JB > Le 26 avr. 2018, à 03:13,

Re: Apache Beam - jenkins question

See https://github.com/apache/beam/blob/master/.test-infra/jenkins/common_job_properties.groovy#L119 for an example of this being done in practice to add the coveralls repo token as an environment variable. On Fri, Apr 27, 2018 at 12:41 PM Jason Kuster wrote: > Hi Kamil,

Re: Custom URNs and runner translation

On Fri, Apr 27, 2018 at 12:34 PM Kenneth Knowles wrote: > On Fri, Apr 27, 2018 at 12:18 PM Thomas Weise wrote: >> The ability to specify with URN and implement custom transforms is also important. Such transforms may not qualify for inclusion in Beam for a

Re: Apache Beam - jenkins question

Hi Kamil, Davor, I think what you want is the Jenkins secrets feature (see https://support.cloudbees.com/hc/en-us/articles/203802500-Injecting-Secrets-into-Jenkins-Build-Jobs). Davor, I believe you are the only one with enough karma on Jenkins to access the credentials UI; once the credential is

Re: Custom URNs and runner translation

On Fri, Apr 27, 2018 at 12:18 PM Thomas Weise wrote: > > The ability to specify with URN and implement custom transforms is also > important. Such transforms may not qualify for inclusion in Beam for a > variety of reasons (only relevant for a specific environment or use case, >

Re: Custom URNs and runner translation

On Fri, Apr 27, 2018 at 12:18 PM Thomas Weise wrote: > Thanks for all the feedback! I agree that the desirable state is to have solid connector implementations for all common integration scenarios as part of Beam. And it seems that the path there would be cross-language IO. >

Re: Custom URNs and runner translation

Thanks for all the feedback! I agree that the desirable state is to have solid connector implementations for all common integration scenarios as part of Beam. And it seems that the path there would be cross-language IO. The ability to specify with URN and implement custom transforms is also

Re: Splittable DoFN in Spark discussion

On Fri, Apr 27, 2018 at 12:06 PM Robert Bradshaw wrote: > On Fri, Apr 27, 2018 at 11:56 AM Kenneth Knowles wrote: > > > I'm still pretty shallow on this topic & this thread, so forgive if I'm > restating or missing things. > > > My understanding is that the

Re: Splittable DoFN in Spark discussion

I'm still pretty shallow on this topic & this thread, so forgive if I'm restating or missing things. My understanding is that the Spark runner does support Beam's triggering semantics for unbounded aggregations, using the same support code from runners/core that all runners use. Relevant code in

Re: Beam SQL Improvements

Can give it a try end of may, sure. (holidays and work constraints will make it hard before). Le 27 avr. 2018 18:26, "Anton Kedin" a écrit : > Romain, > > I don't believe that JSON approach was investigated very thoroughIy. I > mentioned few reasons which will make it not the

Re: Beam SQL Improvements

Romain, I don't believe that JSON approach was investigated very thoroughIy. I mentioned few reasons which will make it not the best choice my opinion, but I may be wrong. Can you put together a design doc or a prototype? Thank you, Anton On Thu, Apr 26, 2018 at 10:17 PM Romain Manni-Bucau

Re: Apache Beam Jenkins Machines Upgrade

This is fantastic. I saw the reduction of the build times in Jenkins today, they reduced dramatically. Excellent work guys congratulations! On Fri, Apr 27, 2018 at 3:00 PM Etienne Chauchot wrote: > Awesome ! Very good news ! Thanks for your work guys ! > Etienne > Le

Re: Custom URNs and runner translation

On Thu, Apr 26, 2018 at 8:38 PM Chamikara Jayalath wrote: > > > On Thu, Apr 26, 2018 at 5:59 PM Eugene Kirpichov > wrote: > >> I agree with Thomas' sentiment that cross-language IO is very important >> because of how much work it takes to produce a

Re: Apache Beam Jenkins Machines Upgrade

Awesome ! Very good news ! Thanks for your work guys ! Etienne Le vendredi 27 avril 2018 à 00:15 +, Eugene Kirpichov a écrit : > This sounds awesome, thanks to everybody involved! > > On Thu, Apr 26, 2018 at 4:28 PM Yifan Zou wrote: > > Greetings, > > > > Most of you

Re: Maven build broken

+1 to what Ismaël and JB said Etienne Le jeudi 26 avril 2018 à 16:25 +, Ismaël Mejía a écrit : > Thanks Thomas for the fix, > Agree 100% maven should still work until the decision to drop it off is > taken. > > On Thu, Apr 26, 2018 at 5:51 PM Thomas Weise wrote: > > > > >

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #21

See Changes: [rober] Move examples to godoc examples part 1. [coheigea] Use Collections.emptyList() instead of Collections.EMPTY_LIST [coheigea] Simplify single-line lambdas [coheigea]