reuvenlax commented on code in PR #40321:
URL: https://github.com/apache/beam/pull/40321#discussion_r4126587444
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java:
##########
@@ -49,6 +56,16 @@ public static GcsStager fromOptions(PipelineOptions options)
{
*/
@Override
public List<DataflowPackage> stageFiles(List<StagedFile> filesToStage) {
+ String stagingLocation = options.getStagingLocation();
+ if (stagingLocation != null &&
TestDataflowRunner.class.equals(options.getRunner())) {
+ List<Object> cacheKey = Arrays.asList(stagingLocation, filesToStage);
Review Comment:
Done
##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/TestDataflowRunnerTest.java:
##########
@@ -102,6 +102,24 @@ public void testToString() {
"TestDataflowRunner#TestAppName",
TestDataflowRunner.fromOptions(options).toString());
}
+ @Test
Review Comment:
it's because when doing this only TestDataflowRunner.fromOptions(options) is
called and never Pipeline.create(options). Pipeline.create(options) registers
the filesystems as a side effect.
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
