Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-26 Thread via GitHub 


scwhittle merged PR #32503:
URL: https://github.com/apache/beam/pull/32503


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-26 Thread via GitHub 


scwhittle commented on PR #32503:
URL: https://github.com/apache/beam/pull/32503#issuecomment-2376629187

   Failure is unrelated pubsub auth error


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-25 Thread via GitHub 


arunpandianp commented on PR #32503:
URL: https://github.com/apache/beam/pull/32503#issuecomment-2375551334

   Run Java Precommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-25 Thread via GitHub 


arunpandianp commented on PR #32503:
URL: https://github.com/apache/beam/pull/32503#issuecomment-2375508750

   Run Java Precommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-25 Thread via GitHub 


arunpandianp commented on code in PR #32503:
URL: https://github.com/apache/beam/pull/32503#discussion_r1776125464


##
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClientTest.java:
##
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc;
+
+import static org.junit.Assert.assertNotSame;
+import static org.junit.Assert.assertSame;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
+import 
org.apache.beam.runners.dataflow.worker.streaming.config.StreamingGlobalConfig;
+import 
org.apache.beam.runners.dataflow.worker.windmill.CloudWindmillServiceV1Alpha1Grpc.CloudWindmillServiceV1Alpha1Stub;
+import 
org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerRunnerV1Settings;
+import 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCachingStubFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactoryFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.vendor.grpc.v1p60p1.io.grpc.testing.GrpcCleanupRule;
+import 
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableSet;
+import 
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Lists;
+import 
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+@RunWith(Enclosed.class)
+public class GrpcDispatcherClientTest {
+
+  static class GrpcDispatcherClientTestBase {
+
+@Rule public final GrpcCleanupRule grpcCleanup = new GrpcCleanupRule();
+
+final ChannelCachingStubFactory stubFactory =
+new FakeWindmillStubFactory(
+() ->
+grpcCleanup.register(
+
WindmillChannelFactory.inProcessChannel("GrpcDispatcherClientTestChannel")));
+
+static StreamingGlobalConfig getGlobalConfig(boolean 
useWindmillIsolatedChannels) {
+  return StreamingGlobalConfig.builder()
+  
.setWindmillServiceEndpoints(ImmutableSet.of(HostAndPort.fromString("windmill:1234")))
+  .setUserWorkerJobSettings(
+  UserWorkerRunnerV1Settings.newBuilder()
+  .setUseWindmillIsolatedChannels(useWindmillIsolatedChannels)
+  .build())
+  .build();
+}
+  }
+
+  @RunWith(JUnit4.class)
+  public static class RespectsJobSettingTest extends 
GrpcDispatcherClientTestBase {
+
+@Test
+public void createsNewStubWhenIsolatedChannelsConfigIsChanged() {
+  DataflowWorkerHarnessOptions options =
+  PipelineOptionsFactory.as(DataflowWorkerHarnessOptions.class);
+  options.setExperiments(
+  Lists.newArrayList(
+  
GrpcDispatcherClient.STREAMING_ENGINE_USE_JOB_SETTINGS_FOR_ISOLATED_CHANNELS));
+  GrpcDispatcherClient dispatcherClient =
+  GrpcDispatcherClient.create(options, new 
FakeWindmillStubFactoryFactory(stubFactory));
+  // Create first time with Isolated channels disabled
+  
dispatcherClient.onJobConfig(getGlobalConfig(/*useWindmillIsolatedChannels=*/ 
false));
+  CloudWindmillServiceV1Alpha1Stub stub1 = 
dispatcherClient.getWindmillServiceStub();
+  CloudWindmillServiceV1Alpha1Stub stub2 = 
dispatcherClient.getWindmillServiceStub();
+  assertSame(stub2, stub1);
+
+  // Enable Isolated channels
+  
dispatcherClient.onJobConfig(getGlobalConfig(/*useWindmillIsolatedChannels=*/ 
true));
+  CloudWindmillServiceV1Alpha1Stub stub3 = 
dispatcherClient.getWindmillServiceStub();

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-25 Thread via GitHub 


arunpandianp commented on code in PR #32503:
URL: https://github.com/apache/beam/pull/32503#discussion_r1776125341


##
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import static 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.remoteChannel;
+
+import com.google.auth.Credentials;
+import java.util.function.Function;
+import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.vendor.grpc.v1p60p1.io.grpc.ManagedChannel;
+
+public class WindmillStubFactoryFactoryImpl implements 
WindmillStubFactoryFactory {
+
+  private final int windmillServiceRpcChannelAliveTimeoutSec;
+  private final Credentials gcpCredential;
+
+  public WindmillStubFactoryFactoryImpl(DataflowWorkerHarnessOptions 
workerOptions) {
+this.gcpCredential = workerOptions.getGcpCredential();
+this.windmillServiceRpcChannelAliveTimeoutSec =
+workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec();
+  }
+
+  @Override
+  public WindmillStubFactory makeWindmillStubFactory(boolean 
useIsolatedChannels) {
+Function channelFactory =
+serviceAddress -> remoteChannel(serviceAddress, 
windmillServiceRpcChannelAliveTimeoutSec);
+ChannelCache channelCache =
+ChannelCache.create(
+serviceAddress ->
+// IsolationChannel will create and manage separate RPC 
channels to the same
+// serviceAddress via calling the channelFactory, else just 
directly return the
+// RPC channel.
+useIsolatedChannels

Review Comment:
   Ack.



##
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClientTest.java:
##
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc;
+
+import static org.junit.Assert.assertNotSame;
+import static org.junit.Assert.assertSame;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
+import 
org.apache.beam.runners.dataflow.worker.streaming.config.StreamingGlobalConfig;
+import 
org.apache.beam.runners.dataflow.worker.windmill.CloudWindmillServiceV1Alpha1Grpc.CloudWindmillServiceV1Alpha1Stub;
+import 
org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerRunnerV1Settings;
+import 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCachingStubFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactoryFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.vendor.grpc.v1p60p1.io.grpc.testing.GrpcCleanupRule;
+import 
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Immutabl

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-25 Thread via GitHub 


scwhittle commented on code in PR #32503:
URL: https://github.com/apache/beam/pull/32503#discussion_r1775194325


##
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClientTest.java:
##
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc;
+
+import static org.junit.Assert.assertNotSame;
+import static org.junit.Assert.assertSame;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
+import 
org.apache.beam.runners.dataflow.worker.streaming.config.StreamingGlobalConfig;
+import 
org.apache.beam.runners.dataflow.worker.windmill.CloudWindmillServiceV1Alpha1Grpc.CloudWindmillServiceV1Alpha1Stub;
+import 
org.apache.beam.runners.dataflow.worker.windmill.Windmill.UserWorkerRunnerV1Settings;
+import 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.ChannelCachingStubFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactory;
+import 
org.apache.beam.runners.dataflow.worker.windmill.testing.FakeWindmillStubFactoryFactory;
+import org.apache.beam.sdk.options.PipelineOptionsFactory;
+import org.apache.beam.vendor.grpc.v1p60p1.io.grpc.testing.GrpcCleanupRule;
+import 
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableSet;
+import 
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Lists;
+import 
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.net.HostAndPort;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.runners.Enclosed;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameter;
+import org.junit.runners.Parameterized.Parameters;
+
+@RunWith(Enclosed.class)
+public class GrpcDispatcherClientTest {
+
+  static class GrpcDispatcherClientTestBase {
+
+@Rule public final GrpcCleanupRule grpcCleanup = new GrpcCleanupRule();
+
+final ChannelCachingStubFactory stubFactory =
+new FakeWindmillStubFactory(
+() ->
+grpcCleanup.register(
+
WindmillChannelFactory.inProcessChannel("GrpcDispatcherClientTestChannel")));
+
+static StreamingGlobalConfig getGlobalConfig(boolean 
useWindmillIsolatedChannels) {
+  return StreamingGlobalConfig.builder()
+  
.setWindmillServiceEndpoints(ImmutableSet.of(HostAndPort.fromString("windmill:1234")))
+  .setUserWorkerJobSettings(
+  UserWorkerRunnerV1Settings.newBuilder()
+  .setUseWindmillIsolatedChannels(useWindmillIsolatedChannels)
+  .build())
+  .build();
+}
+  }
+
+  @RunWith(JUnit4.class)
+  public static class RespectsJobSettingTest extends 
GrpcDispatcherClientTestBase {
+
+@Test
+public void createsNewStubWhenIsolatedChannelsConfigIsChanged() {
+  DataflowWorkerHarnessOptions options =
+  PipelineOptionsFactory.as(DataflowWorkerHarnessOptions.class);
+  options.setExperiments(
+  Lists.newArrayList(
+  
GrpcDispatcherClient.STREAMING_ENGINE_USE_JOB_SETTINGS_FOR_ISOLATED_CHANNELS));
+  GrpcDispatcherClient dispatcherClient =
+  GrpcDispatcherClient.create(options, new 
FakeWindmillStubFactoryFactory(stubFactory));
+  // Create first time with Isolated channels disabled
+  
dispatcherClient.onJobConfig(getGlobalConfig(/*useWindmillIsolatedChannels=*/ 
false));
+  CloudWindmillServiceV1Alpha1Stub stub1 = 
dispatcherClient.getWindmillServiceStub();
+  CloudWindmillServiceV1Alpha1Stub stub2 = 
dispatcherClient.getWindmillServiceStub();
+  assertSame(stub2, stub1);
+
+  // Enable Isolated channels
+  
dispatcherClient.onJobConfig(getGlobalConfig(/*useWindmillIsolatedChannels=*/ 
true));
+  CloudWindmillServiceV1Alpha1Stub stub3 = 
dispatcherClient.getWindmillServiceStub();
+

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-20 Thread via GitHub 


arunpandianp commented on code in PR #32503:
URL: https://github.com/apache/beam/pull/32503#discussion_r1768325884


##
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServerTest.java:
##
@@ -212,7 +216,16 @@ public  ClientCall interceptCall(
 
 this.client =
 GrpcWindmillServer.newApplianceTestInstance(
-inprocessChannel, new FakeWindmillStubFactory(() -> 
(ManagedChannel) inprocessChannel));
+inprocessChannel,
+new WindmillStubFactoryFactory() {

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-20 Thread via GitHub 


arunpandianp commented on code in PR #32503:
URL: https://github.com/apache/beam/pull/32503#discussion_r1768320815


##
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import static 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.remoteChannel;
+
+import com.google.auth.Credentials;
+import java.util.function.Function;
+import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.vendor.grpc.v1p60p1.io.grpc.ManagedChannel;
+
+public class WindmillStubFactoryFactoryImpl implements 
WindmillStubFactoryFactory {
+
+  private final int windmillServiceRpcChannelAliveTimeoutSec;
+  private final Credentials gcpCredential;
+
+  public WindmillStubFactoryFactoryImpl(DataflowWorkerHarnessOptions 
workerOptions) {
+this.gcpCredential = workerOptions.getGcpCredential();
+this.windmillServiceRpcChannelAliveTimeoutSec =
+workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec();
+  }
+
+  @Override
+  public WindmillStubFactory makeWindmillStubFactory(boolean 
useIsolatedChannels) {
+Function channelFactory =
+serviceAddress -> remoteChannel(serviceAddress, 
windmillServiceRpcChannelAliveTimeoutSec);
+ChannelCache channelCache =
+ChannelCache.create(
+serviceAddress ->
+// IsolationChannel will create and manage separate RPC 
channels to the same
+// serviceAddress via calling the channelFactory, else just 
directly return the
+// RPC channel.
+useIsolatedChannels

Review Comment:
   The channels are cached by ChannelCache, so we have to somehow invalidate 
the caches when the setting changes. Current approach creates a second new 
ChannelCache and let the old one expire. Recreating the cache seems simpler 
than having to invalidate channels, track usage and expire them individually. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-20 Thread via GitHub 


scwhittle commented on code in PR #32503:
URL: https://github.com/apache/beam/pull/32503#discussion_r1768253805


##
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClient.java:
##
@@ -62,31 +67,58 @@ public class GrpcDispatcherClient {
   @GuardedBy("this")
   private final Random rand;
 
+  private final WindmillStubFactoryFactory windmillStubFactoryFactory;
+
+  private final AtomicReference windmillStubFactory = new 
AtomicReference<>();
+
+  private final AtomicBoolean useIsolatedChannels = new AtomicBoolean();
+  private final boolean reactToIsolatedChannelsJobSetting;
+
   private GrpcDispatcherClient(
-  WindmillStubFactory windmillStubFactory,
+  DataflowWorkerHarnessOptions options,
+  WindmillStubFactoryFactory windmillStubFactoryFactory,
   DispatcherStubs initialDispatcherStubs,
   Random rand) {
-this.windmillStubFactory = windmillStubFactory;
+this.windmillStubFactoryFactory = windmillStubFactoryFactory;
+if (DataflowRunner.hasExperiment(
+options, STREAMING_ENGINE_USE_JOB_SETTINGS_FOR_ISOLATED_CHANNELS)) {

Review Comment:
   Discussed, leaving as is and will remove once safe. Defaulting to on will 
let us cleanup eventually and have released sdks use preferred behavior.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-20 Thread via GitHub 


scwhittle commented on code in PR #32503:
URL: https://github.com/apache/beam/pull/32503#discussion_r1768210362


##
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/WindmillStubFactoryFactoryImpl.java:
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs;
+
+import static 
org.apache.beam.runners.dataflow.worker.windmill.client.grpc.stubs.WindmillChannelFactory.remoteChannel;
+
+import com.google.auth.Credentials;
+import java.util.function.Function;
+import org.apache.beam.runners.dataflow.options.DataflowWorkerHarnessOptions;
+import org.apache.beam.runners.dataflow.worker.windmill.WindmillServiceAddress;
+import org.apache.beam.vendor.grpc.v1p60p1.io.grpc.ManagedChannel;
+
+public class WindmillStubFactoryFactoryImpl implements 
WindmillStubFactoryFactory {
+
+  private final int windmillServiceRpcChannelAliveTimeoutSec;
+  private final Credentials gcpCredential;
+
+  public WindmillStubFactoryFactoryImpl(DataflowWorkerHarnessOptions 
workerOptions) {
+this.gcpCredential = workerOptions.getGcpCredential();
+this.windmillServiceRpcChannelAliveTimeoutSec =
+workerOptions.getWindmillServiceRpcChannelAliveTimeoutSec();
+  }
+
+  @Override
+  public WindmillStubFactory makeWindmillStubFactory(boolean 
useIsolatedChannels) {
+Function channelFactory =
+serviceAddress -> remoteChannel(serviceAddress, 
windmillServiceRpcChannelAliveTimeoutSec);
+ChannelCache channelCache =
+ChannelCache.create(
+serviceAddress ->
+// IsolationChannel will create and manage separate RPC 
channels to the same
+// serviceAddress via calling the channelFactory, else just 
directly return the
+// RPC channel.
+useIsolatedChannels

Review Comment:
   would it be simpler to just have IsolatedChannel accept a atomic boolean 
that it then examines on whether or not it should perform the isolation or not?
   
   It seems like we wouldn't have to have as much of the plumbing or the 
factoryfactory if we just always created an IsolatedChannel and just had a way 
to disable it's effect.



##
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcWindmillServerTest.java:
##
@@ -212,7 +216,16 @@ public  ClientCall interceptCall(
 
 this.client =
 GrpcWindmillServer.newApplianceTestInstance(
-inprocessChannel, new FakeWindmillStubFactory(() -> 
(ManagedChannel) inprocessChannel));
+inprocessChannel,
+new WindmillStubFactoryFactory() {

Review Comment:
   use FakeWindmillStubFactoryFactory?



##
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/GrpcDispatcherClient.java:
##
@@ -62,31 +67,58 @@ public class GrpcDispatcherClient {
   @GuardedBy("this")
   private final Random rand;
 
+  private final WindmillStubFactoryFactory windmillStubFactoryFactory;
+
+  private final AtomicReference windmillStubFactory = new 
AtomicReference<>();
+
+  private final AtomicBoolean useIsolatedChannels = new AtomicBoolean();
+  private final boolean reactToIsolatedChannelsJobSetting;
+
   private GrpcDispatcherClient(
-  WindmillStubFactory windmillStubFactory,
+  DataflowWorkerHarnessOptions options,
+  WindmillStubFactoryFactory windmillStubFactoryFactory,
   DispatcherStubs initialDispatcherStubs,
   Random rand) {
-this.windmillStubFactory = windmillStubFactory;
+this.windmillStubFactoryFactory = windmillStubFactoryFactory;
+if (DataflowRunner.hasExperiment(
+options, STREAMING_ENGINE_USE_JOB_SETTINGS_FOR_ISOLATED_CHANNELS)) {

Review Comment:
   why do we need experiment in addition to the job setting?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-20 Thread via GitHub 


arunpandianp commented on PR #32503:
URL: https://github.com/apache/beam/pull/32503#issuecomment-2363160305

   Run Java Precommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-19 Thread via GitHub 


github-actions[bot] commented on PR #32503:
URL: https://github.com/apache/beam/pull/32503#issuecomment-2362858644

   Stopping reviewer notifications for this pull request: review requested by 
someone other than the bot, ceding control. If you'd like to restart, comment 
`assign set of reviewers`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] [Dataflow Streaming] Use isolated windmill streams based on job settings [beam]

2024-09-19 Thread via GitHub 


arunpandianp commented on PR #32503:
URL: https://github.com/apache/beam/pull/32503#issuecomment-2362857078

   R: @scwhittle 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@beam.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org