Re: Virtualenv setup issues on new machine

Not sure, dog seems to be the only meaningful difference in 2 scenarios. On Thu, Feb 28, 2019 at 8:04 PM Udi Meiri wrote: > Weird. Is that a known bug? > > On Thu, Feb 28, 2019 at 3:19 PM Ankur Goenka wrote: > >> The issue seems to be with "." in the virtualenv path. >> virtualenv works after m

Re: Virtualenv setup issues on new machine

Weird. Is that a known bug? On Thu, Feb 28, 2019 at 3:19 PM Ankur Goenka wrote: > The issue seems to be with "." in the virtualenv path. > virtualenv works after moving from > "/usr/local/google/home/goenka/.local/bin" to "/usr/bin" > > On Thu, Feb 28, 2019 at 2:57 PM Udi Meiri wrote: > >> I th

Re: CVE audit gradle plugin

Thank you, I agree this is very important. Does anyone know a similar tool for python and go? On Thu, Feb 28, 2019 at 8:26 AM Etienne Chauchot wrote: > Hi guys, > > I came by this [1] gradle plugin that is a client to the Sonatype OSS > Index CVE database. > > I have set it up here in a branch [

Re: [ANNOUNCE] New committer announcement: Michael Luckey

Congrats Michael! On Thu, Feb 28, 2019 at 3:12 AM Maximilian Michels wrote: > Welcome, it's great to have you onboard Michael! > > On 28.02.19 11:46, Michael Luckey wrote: > > Thanks to all of you for the warm welcome. Really happy to be part of > > this great community! > > > > michel > > > > O

Re: Virtualenv setup issues on new machine

The issue seems to be with "." in the virtualenv path. virtualenv works after moving from "/usr/local/google/home/goenka/.local/bin" to "/usr/bin" On Thu, Feb 28, 2019 at 2:57 PM Udi Meiri wrote: > I think gradle is complaining that the path can't be found. > Is there more information if you run

Re: KafkaIO Exactly-Once & Flink Runner

On Thu, Feb 28, 2019 at 2:42 PM Raghu Angadi wrote: > On Thu, Feb 28, 2019 at 2:34 PM Kenneth Knowles wrote: > >> I'm not sure what a hard fail is. I probably have a shallow >> understanding, but doesn't @RequiresStableInput work for 2PC? The >> preCommit() phase should establish the transaction

Re: Virtualenv setup issues on new machine

I think gradle is complaining that the path can't be found. Is there more information if you run it with --info? On Thu, Feb 28, 2019, 14:35 Ankur Goenka wrote: > Hi Beamers, > > I am trying build python sdk from a fresh git checkout on a new linux > machine but the setupVirtualEnv task is faili

Re: KafkaIO Exactly-Once & Flink Runner

On Thu, Feb 28, 2019 at 2:34 PM Kenneth Knowles wrote: > I'm not sure what a hard fail is. I probably have a shallow understanding, > but doesn't @RequiresStableInput work for 2PC? The preCommit() phase should > establish the transaction and commit() is not called until after checkpoint > finaliz

Virtualenv setup issues on new machine

Hi Beamers, I am trying build python sdk from a fresh git checkout on a new linux machine but the setupVirtualEnv task is failing with the error below. The complete build scan is at https://scans.gradle.com/s/h3jwzeg5aralk/failure?openFailures=WzBd&openStackTraces=WzQsM10#top=0 >From the error it

Re: KafkaIO Exactly-Once & Flink Runner

I'm not sure what a hard fail is. I probably have a shallow understanding, but doesn't @RequiresStableInput work for 2PC? The preCommit() phase should establish the transaction and commit() is not called until after checkpoint finalization. Can you describe the way that it does not work a little bi

Re: KafkaIO Exactly-Once & Flink Runner

On Thu, Feb 28, 2019 at 11:01 AM Kenneth Knowles wrote: > I believe the way you would implement the logic behind Flink's > KafkaProducer would be to have two steps: > > 1. Start transaction > 2. @RequiresStableInput Close transaction > I see. What happens if closing the transaction fails in (2)

Re: KafkaIO Exactly-Once & Flink Runner

I believe the way you would implement the logic behind Flink's KafkaProducer would be to have two steps: 1. Start transaction 2. @RequiresStableInput Close transaction The FlinkRunner would need to insert the "wait until checkpoint finalization" logic wherever it sees @RequiresStableInput, which

Re: KafkaIO Exactly-Once & Flink Runner

On Thu, Feb 28, 2019 at 10:41 AM Raghu Angadi wrote: > > Now why does the Flink Runner not support KafkaIO EOS? Flink's native >> KafkaProducer supports exactly-once. It simply commits the pending >> transaction once it has completed a checkpoint. > > > > On Thu, Feb 28, 2019 at 9:59 AM Maximilia

Re: KafkaIO Exactly-Once & Flink Runner

> Now why does the Flink Runner not support KafkaIO EOS? Flink's native > KafkaProducer supports exactly-once. It simply commits the pending > transaction once it has completed a checkpoint. On Thu, Feb 28, 2019 at 9:59 AM Maximilian Michels wrote: > Hi, > > I came across KafkaIO's Runner whit

Re: KafkaIO Exactly-Once & Flink Runner

This is exactly what RequiresStableInput is supposed to do. On the Flink runner, this would be implemented by delaying processing until the current checkpoint is done . In fact many sinks are probably subtly broken on the Flink runner today without RequiresStableInput, so we really need to finish t

KafkaIO Exactly-Once & Flink Runner

Hi, I came across KafkaIO's Runner whitelist [1] for enabling exactly-once semantics (EOS). I think it is questionable to exclude Runners from inside a transform, but I see that the intention was to save users from surprises. Now why does the Flink Runner not support KafkaIO EOS? Flink's nat

CVE audit gradle plugin

Hi guys, I came by this [1] gradle plugin that is a client to the Sonatype OSS Index CVE database. I have set it up here in a branch [2], though the cache is not configured and the number of requests is limited. It can be run with "gradle --info audit" It could be nice to have something like t

Re: Merge of vendored Guava (Some PRs need a rebase)

If someone is using BigTableIO with bigtable-client-core then having BigTableIO and bigtable-client-core both depend on Guava 26.0 is fine, right? Specifically, a user of BigTableIO after https://github.com/apache/beam/pull/7957 will still have non-vendored Guava on the classpath due to the transit

Re: [ANNOUNCE] New committer announcement: Michael Luckey

Welcome, it's great to have you onboard Michael! On 28.02.19 11:46, Michael Luckey wrote: Thanks to all of you for the warm welcome. Really happy to be part of this great community! michel On Thu, Feb 28, 2019 at 8:39 AM David Morávek > wrote: Congrats Mi

Re: [ANNOUNCE] New committer announcement: Michael Luckey

Thanks to all of you for the warm welcome. Really happy to be part of this great community! michel On Thu, Feb 28, 2019 at 8:39 AM David Morávek wrote: > Congrats Michael! 🍾 > > D. > > > On 28 Feb 2019, at 03:27, Ismaël Mejía wrote: > > > > Congratulations Michael, and thanks for all the contr

Re: [ANNOUNCE] New committer announcement: Michael Luckey

Congrats and welcome! :) czw., 28 lut 2019 o 08:39 David Morávek napisał(a): > Congrats Michael! 🍾 > > D. > > > On 28 Feb 2019, at 03:27, Ismaël Mejía wrote: > > > > Congratulations Michael, and thanks for all the contributions! > > > >> On Wed, Feb 27, 2019 at 6:30 PM Ankur Goenka wrote: > >>

Re: Merge of vendored Guava (Some PRs need a rebase)

For the past week, two independent people have asked me if I can help with guava NoSuchMethodError in BigtableIO. It turns out we still have a potential problem with dependencies that don't vendor guava, in this case, it was bigtable-client-core that depends on guava-26.0. However, the root cause o