Flaky test issue report

This is your daily summary of Beam's current flaky tests. These are P1 issues because they have a major negative impact on the community and make it hard to determine the quality of the software. BEAM-12250: Java ValidatesRunner Postcommits timing out (https://issues.apache.org/jira/browse/

P1 issues report

This is your daily summary of Beam's current P1 issues, not including flaky tests. See https://beam.apache.org/contribute/jira-priorities/#p1-critical for the meaning and expectations around P1 issues. BEAM-12282: Failure in ':vendor:grpc-1_26_0:validateVendoring' (https://issues.apache.or

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

When you use cross-language Java transforms from Python we use the default environment for Java transforms which always gets set to Docker. https://github.com/apache/beam/blob/7f0d11e65bbcd3e9c565a50aa6a56c0631c4358b/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/Ex

Re: Looking to Contribute

Hi Zach! I've added you as contributor. Welcome! Best -P. On Tue, May 4, 2021 at 12:50 PM Zachary Houfek wrote: > Hi, I'm a new member of the Cloud Dataflow team and will be looking to > contribute to Beam in the near future. > > My Jira Id is zhoufek > > Thanks, > Zach > > -- > > Zachary Houfek

Looking to Contribute

Hi, I'm a new member of the Cloud Dataflow team and will be looking to contribute to Beam in the near future. My Jira Id is zhoufek Thanks, Zach -- Zachary Houfek Software Engineer DataPLS PLAT zhou...@google.com (650) 695-8003

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

But looking at the code of the exception it seems that it tries to use docker only because it thinks it's in a docker environment, no? S

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Ah, I think you need the DOCKER environment to use cross-language transforms not the EXTERNAL environment (agree that the terminology is confusing). On Tue, May 4, 2021 at 11:04 AM Nir Gazit wrote: > Yes that’s on purpose. I’m running in Kubernetes which makes it hard to > install docker on the

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Yes that’s on purpose. I’m running in Kubernetes which makes it hard to install docker on the pods so I don’t want to use the docker environment. That’s why I specified EXTERNAL environment in PipelineOptions. However, it seems that it doesn’t get propagated. On Tue, 4 May 2021 at 20:59 Chamikara

Re: Potential bug with Kafka (or other external) IO in Python Portable Runner

Is it possible that you don't have the "docker" command available in your system ? On Tue, May 4, 2021 at 10:28 AM Nir Gazit wrote: > Hey, > I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL > environment. However, when the pipeline is run, the error below is thrown, > which im

Potential bug with Kafka (or other external) IO in Python Portable Runner

Hey, I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL environment. However, when the pipeline is run, the error below is thrown, which implies that for some reason the external environment pipeline options didn't get in. When replacing the Kafka Source with an S3 source (for exam

Re: Timer.withOutputTimestamp().offset().setRelative seems unusable with event time

Mean to also add +Reuven Lax On Tue, May 4, 2021 at 9:41 AM Kenneth Knowles wrote: > Explicitly pinging a couple folks who were involved in the original change > which yours essentially reverts. There's a model question here that I want > to clarify on-list: > > When you have a ParDo setting ti

Re: Timer.withOutputTimestamp().offset().setRelative seems unusable with event time

Explicitly pinging a couple folks who were involved in the original change which yours essentially reverts. There's a model question here that I want to clarify on-list: When you have a ParDo setting timers, you have an additional watermark that must be considered: - input element watermark - o

RE: [PROPOSAL] Upgrade Cassandra driver from 3.x to 4.x in CassandraIO

Hi All, Thanks for your valuable comments. We will work on these and get back to you. Regards Anup From: Ismaël Mejía Sent: Friday, April 30, 2021 6:44 PM To: dev Subject: Re: [PROPOSAL] Upgrade Cassandra driver from 3.x to 4.x in CassandraIO Hello, My excuses for not having commented on thi

Re: Timer.withOutputTimestamp().offset().setRelative seems unusable with event time

Hi Kenn, I created BEAM-12276 [1] with PR [2].  Jan [1] https://issues.apache.org/jira/browse/BEAM-12276 [2] https://github.com/apache/beam/pull/14718 On 5/3/21 7:46 PM, Kenneth Knowles wrote: This seems like just a bug. If you set a timer for X and have output timestamp Y where X < Y this s