shunping commented on code in PR #40713:
URL: https://github.com/apache/beam/pull/40713#discussion_r4352796098
##########
sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java:
##########
@@ -127,16 +127,41 @@ public static boolean isWildcard(GcsPath spec) {
return GcsPath.isWildcard(spec);
}
+ /**
+ * Experiment that falls back to GcsUtil V1 (gcsio). V2 (java-storage) is
the default, and this
+ * experiment is the escape hatch should V2 misbehave for a pipeline.
+ */
+ public static final String DISABLE_GCSUTIL_V2_EXPERIMENT =
"disable_gcsutil_v2";
+
+ /**
+ * Experiment that used to opt into GcsUtil V2. V2 is now the default, so
this is a no-op kept so
+ * that existing pipelines that still pass it keep working. {@link
#DISABLE_GCSUTIL_V2_EXPERIMENT}
+ * takes precedence if both are set.
+ */
+ public static final String USE_GCSUTIL_V2_EXPERIMENT = "use_gcsutil_v2";
+
+ /**
+ * Experiment that makes GcsUtil V1 talk to GCS over gRPC. Only V1 (gcsio)
supports it, so it also
+ * falls back to V1, the same as {@link #DISABLE_GCSUTIL_V2_EXPERIMENT}.
+ */
+ public static final String USE_GRPC_FOR_GCS_EXPERIMENT = "use_grpc_for_gcs";
+
GcsUtil(PipelineOptions options) {
this.delegate = new GcsUtilV1.GcsUtilFactory().create(options);
- if (ExperimentalOptions.hasExperiment(options, "use_gcsutil_v2")) {
+ @Nullable String v1Experiment = null;
+ if (ExperimentalOptions.hasExperiment(options,
DISABLE_GCSUTIL_V2_EXPERIMENT)) {
+ v1Experiment = DISABLE_GCSUTIL_V2_EXPERIMENT;
+ } else if (ExperimentalOptions.hasExperiment(options,
USE_GRPC_FOR_GCS_EXPERIMENT)) {
+ v1Experiment = USE_GRPC_FOR_GCS_EXPERIMENT;
Review Comment:
It is in our plan to support gRPC in V2. However, since gRPC support in V1
is still controlled by an experiment, we didn't want it to block the default
switch to V2.
In addition, we'll need to run another round of benchmarks comparing
HTTP/gRPC modes on V2 and gRPC mode on V1/V2 to make sure there are no
discrepancies before we route gRPC support to V2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
