This is an automated email from the ASF dual-hosted git repository.
herohde pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from f67541e Merge pull request #6077: [BEAM-4862] Fixes bug in Spanner's
MutationGroupEncoder by converting timestamps into Long and not Int.
add 89fe68a Add utilities for manipulating proto pipelines
add 4224bcc [BEAM-4636] Refactor Go dataflow runner for more flexible uses
add 585fdce [BEAM-4813] Switch to translation from model protos to
Dataflow v1beta3
add 1abd375 Add more Go integration tests and fix CoGBK bugs
add dc9be5e [BEAM-4636] Ensure unique filename for compiled worker
binaries
add 6228ebd Handle explicit flatten
add b6433a3 Fix Dataflow argument bug
add bdaefa6 CR: tweak code and fix comments
new dc1111b [BEAM-4813] Refactor Go Dataflow runner and translation
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
sdks/go/pkg/beam/core/runtime/exec/plan.go | 10 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 3 +
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 301 +++++++++++++
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 266 ------------
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 184 ++++----
sdks/go/pkg/beam/core/runtime/graphx/tree.go | 31 --
sdks/go/pkg/beam/core/runtime/pipelinex/clone.go | 56 +++
.../runtime/pipelinex/clone_test.go} | 32 +-
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 206 +++++++++
.../beam/core/runtime/pipelinex/replace_test.go | 119 +++++
sdks/go/pkg/beam/core/runtime/pipelinex/util.go | 89 ++++
sdks/go/pkg/beam/core/util/reflectx/util.go | 92 ++++
sdks/go/pkg/beam/core/util/reflectx/util_test.go | 106 +++++
.../beam/core/util/stringx/{bytes.go => map.go} | 40 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 286 ++----------
.../beam/runners/dataflow/dataflowlib/execute.go | 110 +++++
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 220 ++++++++++
.../runners/dataflow/{ => dataflowlib}/messages.go | 2 +-
.../pkg/beam/runners/dataflow/dataflowlib/stage.go | 56 +++
.../beam/runners/dataflow/dataflowlib/translate.go | 347 +++++++++++++++
sdks/go/pkg/beam/runners/dataflow/translate.go | 479 ---------------------
.../beam/runners/universal/runnerlib/compile.go | 7 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 2 +
sdks/go/pkg/beam/runners/universal/universal.go | 6 +-
.../pkg/beam/testing/passert/{hash.go => sum.go} | 44 +-
sdks/go/pkg/beam/util/gcsx/gcs.go | 12 +
sdks/go/test/integration/driver.go | 11 +-
sdks/go/test/integration/primitives/cogbk.go | 84 ++++
.../integration/primitives/cogbk_test.go} | 19 +-
.../integration/primitives/flatten.go} | 40 +-
.../primitives/flatten_test.go} | 10 +-
sdks/go/test/integration/primitives/pardo.go | 82 ++++
.../primitives}/pardo_test.go | 16 +-
sdks/go/test/regression/pardo.go | 80 +++-
sdks/go/test/regression/pardo_test.go | 24 ++
35 files changed, 2240 insertions(+), 1232 deletions(-)
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/dataflow.go
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/clone.go
copy sdks/go/pkg/beam/{coder_test.go => core/runtime/pipelinex/clone_test.go}
(62%)
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/replace.go
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/replace_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/pipelinex/util.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/util.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/util_test.go
copy sdks/go/pkg/beam/core/util/stringx/{bytes.go => map.go} (50%)
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/execute.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
rename sdks/go/pkg/beam/runners/dataflow/{ => dataflowlib}/messages.go (99%)
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/stage.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/translate.go
delete mode 100644 sdks/go/pkg/beam/runners/dataflow/translate.go
copy sdks/go/pkg/beam/testing/passert/{hash.go => sum.go} (50%)
create mode 100644 sdks/go/test/integration/primitives/cogbk.go
copy sdks/go/{pkg/beam/core/runtime/exec/status.go =>
test/integration/primitives/cogbk_test.go} (80%)
copy sdks/go/{pkg/beam/transforms/filter/distinct.go =>
test/integration/primitives/flatten.go} (57%)
copy sdks/go/test/{regression/pardo_test.go =>
integration/primitives/flatten_test.go} (80%)
create mode 100644 sdks/go/test/integration/primitives/pardo.go
copy sdks/go/test/{regression => integration/primitives}/pardo_test.go (73%)
