y1chi commented on code in PR #29230:
URL: https://github.com/apache/beam/pull/29230#discussion_r1378221242
##########
sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java:
##########
@@ -477,6 +480,39 @@ public void testBuildWrite() {
assertThat(write.getComputeNumShards(), equalTo(originalSharding));
}
+ @Test
+ public void testWriteTransformName() {
+ PipelineOptions options = TestPipeline.testingPipelineOptions();
+ Pipeline pipeline = Pipeline.create(options);
+
+ SimpleSink<Void> sink = makeSimpleSink();
+ WriteFiles<String, ?, String> write =
+
WriteFiles.to(sink).withNumShards(0).withWindowedWrites().withRunnerDeterminedSharding();
+
+ List<String> inputs = Lists.newArrayList();
+ for (int i = 0; i < 100; ++i) {
+ inputs.add("mambo_number_" + i);
+ }
+
+ // Prepare timestamps for the elements.
+ List<Long> timestamps = new ArrayList<>();
+ for (long i = 0; i < inputs.size(); i++) {
+ timestamps.add(i + 1);
+ }
+
+ assertThat((SimpleSink<Void>) write.getSink(), is(sink));
+ assertThat(write.getComputeNumShards(), nullValue());
+
+ pipeline
+ .apply(Create.timestamped(inputs,
timestamps).withCoder(StringUtf8Coder.of()))
+ .setIsBoundedInternal(IsBounded.UNBOUNDED)
+ .apply(Window.into(FixedWindows.of(Duration.millis(1))))
+ .apply(write);
+ // This should not change, this ensures that the transform name is what
the dataflow runner
+ // expects.
+ logged.verifyDebug("WriteAutoShardedBundlesToTempFiles");
Review Comment:
Can we test more directly? For example we may be able to get the step name
from translated job specification
https://github.com/apache/beam/blob/d329a7e9b85d729a474395abfe1975c0f3649550/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java#L199
just in case the name is somehow changed or removed after translation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
