This closes #3977: [BEAM-2963] Stage pipeline in DataflowRunner Remove duplicate mocking in DataflowRunnerTest Add assertion that valid jobs must have staged pipeline Stage the pipeline without using a temp file Stage the portable pipeline in Dataflow Add ability to stage explicit file list Improve GcsFileSystem errors messages slightly
Project: http://git-wip-us.apache.org/repos/asf/beam/repo Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1974b920 Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1974b920 Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1974b920 Branch: refs/heads/master Commit: 1974b920e4b3bbe8549e25fe789f9dada13c1769 Parents: 2acdc74 128f3a6 Author: Kenneth Knowles <[email protected]> Authored: Wed Oct 18 13:08:50 2017 -0700 Committer: Kenneth Knowles <[email protected]> Committed: Wed Oct 18 13:08:50 2017 -0700 ---------------------------------------------------------------------- .../beam/runners/dataflow/DataflowRunner.java | 15 ++- .../beam/runners/dataflow/util/GcsStager.java | 53 +++++++-- .../beam/runners/dataflow/util/PackageUtil.java | 116 ++++++++++++++----- .../beam/runners/dataflow/util/Stager.java | 32 ++++- .../runners/dataflow/DataflowRunnerTest.java | 87 +++++++++++--- .../extensions/gcp/storage/GcsFileSystem.java | 5 +- 6 files changed, 239 insertions(+), 69 deletions(-) ----------------------------------------------------------------------
