Re: Jenkins CI currently unavailable

2022-06-13 Thread Robert Burke
There are a couple of Go SDK PRs that are basically blocked on final manual runs of the post commits, that we'd like to get in for the 2.40 cut. Are we intending on delaying the 2.40 cut a little bit so PRs like those can make it in? On Mon, Jun 13, 2022, 1:32 PM Ahmet Altay wrote: > Thank

Re: Jenkins CI currently unavailable

2022-06-13 Thread Robert Burke
You know, I do forget that committers can manually trigger Jenkins runs. And after fiddling with the Jenkins options and filling in the expected, but missing PR number parameter i think I've managed it. Thanks for the reminder! On Mon, Jun 13, 2022, 3:38 PM Kiley Sok wrote: > Can you run the

Re: Jenkins CI currently unavailable

2022-06-13 Thread Robert Burke
Logged in as a Committer, You need to open the configuration menu for the task, add in a string parameter for 'ghprPullId'. Save. Then you can go to Build With Parameters, populate the new field with the PR number And the sha1 with origin/pr//merge The you can kick off the job. It doesn't

Re: Jenkins CI currently unavailable

2022-06-13 Thread Kenneth Knowles
I just also want to say that you can run the tests locally and have people believe you when you say they passed. It isn't as good as CI but it is better than nothing. I am referring to the case where you want to manually add a postcommit status to your PR. If you orchestrate with gradle then you

Re: Beam Java DoFn Performance (and Auto-Vectorization)

2022-06-13 Thread Robert Burke
Fascinating! I ran through a similar exercise for the Go SDK when I was fixing the batch Load Tests, but optimizing some specialized DoFn execution nodes calls for DoFns that don't observe windows, or return For comparison, the Go SDK already has a Returning DoFn as you call it, but I found that

Re: Join a meeting to help coordinate implementing a Dask Runner for Beam

2022-06-13 Thread Kenneth Knowles
I would love to try to join any meetings if you add me. My calendar is too chaotic to be useful on the when2meet :-) but I can often move things around. Kenn On Wed, Jun 8, 2022 at 2:50 PM Brian Hulette wrote: > Thanks for reaching out, Ryan, this sounds really cool. I added my > availability

Chained Job Graph Apache Beam | Dataflow

2022-06-13 Thread Ravi Kapoor
Hi Team, I am currently using Beam in my project with Dataflow Runner. I am trying to create a pipeline where the data flows from the source to staging then to target such as: A (Source) -> B(Staging) -> C (Target) When I create a pipeline as below: PCollection table_A_records =

Re: Jenkins CI currently unavailable

2022-06-13 Thread Kenneth Knowles
I definitely think we should freeze until we have precommits re-enabled. We have a hypothesis that https://github.com/apache/beam/pull/21821 may reduce the impact on the Jenkins master. We would lose phrase triggering on postcommits, but we would still have it on precommits for retriggering after

Re: Jenkins CI currently unavailable

2022-06-13 Thread Kenneth Knowles
Yes, the ghprb plugin was disabled. That was the entire action. I believe my PR will reduce the load caused by the ghprb plugin; we are currently restarting Jenkins to re-enable it. So we can unfreeze master as soon as Jenkins reboots. Basically, if your PR has a precommit status great, otherwise