This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from d086d87 Merge pull request #4677: Add a Beam Jenkins Runner Inventory
Cron
add e2aa9b3 Initial sketches of a Go SDK
add 7197d5e Initial version of the direct style w/ direct runner.
Incomplete.
add e92e919 Add Data as UserFn context w/ immediate value.
add 995b2ec Added no-I/O wordcount for profiling.
add 900da2e Fleshed out possible approach to generic transformations.
add 7bc0498 Add “dag” example that use multiplexing and side input.
add 915416a Added a more complex DAG example.
add fa8723a Add yatzy example with more complex construction-time setup
add 60d17c1 Add proto for Fn API
add b1aaac5 Add beam.Composite helper for the most common pattern to
align with java
add d08fdaa Move pipeline-construction time errors into an accumulator
add ee71e83 Add Dataflow job and Fn API clients. Incomplete.
add 61db2fb Add binary cross-compile and upload to Dataflow runner.
Incomplete.
add 5ce4d1c Add tentative runner indirection (default: local).
add 01111a9 Made data flow runner detect user main for cross-compilation.
add 08ba947 Remove error accumulation in favor of panic.
add f66bd66 Improve Dataflow translation of coders, side input and
composite names.
add c579e85 Fix name for AsView.
add 26a99d5 Use 2 grpc endpoints in harness
add d5cf63d Add gRPC harness logging
add 0b2fdbc Flesh out harness and serialization further.
add 6f4af7f Made the dataflow runner wait for job termination by default
add 3c8f986 beam:
add 687cbf8 beam:
add c47a546 combinefn.go: fix compilation issues
add b9d03b2 Improve dataflow serialization and execution. Incomplete.
add af65f9d Sleep 30 sec in wordcap to allow logs to propagate to Cloud
Logging.
add 38ddbbb Move the 30s sleep for logging to the harness instead of in
WordCap.
add e185f48 Post-review updates.
add 8dcd0c4 Doc updates.
add d64868c Flesh out coders. Incomplete.
add ca49202 Added prototype implementation of more coders and the runner
source. Incomplete.
add 0444a45 dofn: illustrates how dofns are written.
add 9e2a7cc beam: add viewfn and windowfn to side inputs match support
Beam 1.0
add b767a95 dofn: timers
add be7e061 Complete revamp: coders, graph and execution use element-wise
processing.
add 055c859 Fix coder encoding for Dataflow side input. Otherwise, the
job is rejected.
add 22536e3 Added more godoc comments to graph types.
add 747930f Added more comments plus made local GBK use coder equality.
add ce8c4df Added Flatten support and “forest” example that uses it.
add 61d8932 Move bigqueryio to defunct
add ca59228 Make forest example print less
add dbafc8b Add external struct registry and serialization.
add 174a2b0 Updated comments in node.go.
add 72e3bbe Replace real type with 'full type' since that's the current
term.
add 58dda52 Refactor Fn API dependency.
add bf6ff10 Added more comments to the runner/dataflow and
runner/beamexec packages
add bcc5001 Fix most go vet issues
add 24e2078 Make core operations panic to cut down on the error
propagation boilerplate
add 2662fc8 Add more comments to the graph package.
add 4eab543 Add DoFn wrapper to handle either function or (ptr to) struct
add eca6fec Fix remaining go vet warnings.
add f1953d3 Code review for beam/graph/coder package.
add 5951d28 Code review of the runtime/graphx package.
add 978b60e Remove Data options in favor of using a Fn struct
add d56db10 Code review of the beam/graph/userfn package.
add 9a74200 Code review for beam/graph package.
add 15489ee godoc for runtime/graphx
add af323c8 Add support for []T and Combine functions
add b1d8df2 Add adapted documentation from the Java SDK to the beam
package
add 4f5126c Update snapshot of Fn API.
add bf3cac1 Add experiments flag to the Dataflow runner
add 3c848b6 Remove context arg from beamexec.Init
add ca38035 Migration to Runner API.
add 7b6ca59 Add support for creating DOT graphs.
add 0dc6e2d Make pretty printing of types and coders more concise
add a854fbc Add flexible Signature to aid type checking
add c87e176 Adding unit testability to harness translation.
add b3f65ea Fix crash due to initialization order
add ea6c7c4 Add CreateValues and Impulse
add 1637568 Add Runner API support for WindowingStrategy.
add 65d0195 Run goimports on baseline.
add dc23901 Fix encoding of global window strategy.
add adc7869 Ensure the windowed value is atomically encoded.
add ce880b7 Limit gRPC messages to max size.
add 841520e Developer conveniences for running jobs.
add 35ff36e Fix sends to not close the network channel.
add 461c315 Add re-iterable side input
add ea78889 Add per-key Combine
add 121acd8 Add Min
add cbd622e Reorganize non-user-facing code into core
add c0962fb Make type register reject unnamed or predeclared types
add 6b51242 Add type specialization tool
add 0d439e9 Don't run grpc plugin in generate phase.
add 40d8111 Fix import reference path for runner API proto.
add 34d9647 Revamp runner registration as _ imports
add a9136b3 Add stats.Max and Mean
add 214c076 Add global pipeline options
add 11f9c03 Unify global and per-key combiners
add c0658c5 Add beam convenience wrapper for imports and runner selection
add dca064e Add session recording and CPU profiling to harness.
add d17c601 Add ptest and passert for testing pipelines
add 9e575d6 Add GCS and glob support to textio
add 4fa8db0 Add BigQuery IO and examples
add f01e74a Adds a session runner for testing.
add 1627288 Add Partition and dynamic functions
add 9bf757f Adding example that returns 10 words that contain provided
search string.
add e95172b Remove duplicate LOG line
add 5322ed6 Enable Combine Fns in Dataflow runner by modifying
translation.
add 5e26b7a Fixing type bug by dropping T and using underlying type of
value in addInput(). Changing Count() to use Sum().
add dbbe214 Adding graph validation at build time.
add fc5ab4f Import the Fn API changes.
add f945954 Simple changes to support new Fn API coder changes.
add bf6bc20 Update translator to work with new Fn API changes.
add 5c9cba6 Use appropriate equality tests.
add b897ea4 Fix test to not use path of package.
add 330d037 Renaming directory to match package name.
add 895b735 Fixing random nits in comments.
add 308fee0 Modify build command to avoid bash.
add c762062 Fixing selected golint issues.
add f109f5e Addressing import review comments.
add 0c89de4 Add coder specialization for bytes/strings.
add 7ee4c51 Adding unit tests to stats.
add 00fefa5 Fixing typo.
add b7b8c31 Add beam.External
add da1c077 Fix grpc.Dial calls to block properly.
add 3246480 Creates a symtab verifier by running Sym2Addr and Addr2Sym in
a binary.
add c75f747 Add spec field to help interpretation of payload.
add 7ac34c7 Use beam.T alias for typex.T etc with Go 1.9
add a1ed07c Move shared GCP options to a separate package
add fb2c2a3 Update portability protos
add 3fdf71e Remove old source/sink from beam package
add e6e05f5 Add context-aware logging for both pipeline-submission time
and runtime
add 52fdf4b Fix coder inference for strings.
add 5f38b6d Improve tornadoes example
add 6187549 Fix beam.External to map arguments correctly.
add e99cb8e Added comments to yatzy and forest
add e0a5b71 Add comments to tornadoes from the java counterpart
add 738415f Rename Pipeline Composite to Scope
add b8616e8 Add 3 progressive wordcount examples
add 10f9c28 Clarify comments in wordcount pipelines
add 55ea735 Add apache 2.0 license to files
add c665314 Updates to examples.
add afc6097 Adding more godoc for the main beam package.
add 00a3bbc Update to new proto structure
add bc58989 Split Combine and fields in to global and per-key variants
add 1899718 Refactor Flatten of a single []T into Explode
add bd962f6 Rename local runner to direct runner
add 663046e Fix argument index error in ParDo execution
add 670b13c Add Apache copyright header to files that need it.
add 2d5a991 Made debug.Head not just work per bundle
add 47b2e31 Impose a total ordering on Fn parameters.
add 0b0aeac Rename Dedup to Distinct for consistency with other SDKs
add baa38c2 Add coder to model coder translation
add a088674 Simplify harness coder translation
add 663fad3 Split Pipeline into Pipeline and Scope
add 5743a37 Relocate Go SDK code
add 999985c Fix Go SDK maven build
add 38df1b7 Move Go SKD to latest version of bigquery
add 473d7a0 Merge pull request #4200 from herohde/go-sdk2
add 62de5ca Add Go SDK container image
add 06f6411 Merge pull request #4215: [BEAM-3287] Add Go SDK container
image
add c2c152a Add Go SDK README
add 688782c Merge pull request #4217: [BEAM-2083] Add Go SDK README
add 7c958a8 Update version for Go Dataflow pipelines
add e966d4e Make Scope a value type
add 008ba51 Merge pull request #4230: [BEAM-2083] Make Scope a value type
in Go SDK
add fffaed1 Add Go graph/pipeline translation
add f42ab80 Stage Go model pipeline for Dataflow
add eb4367d Use pipeline unmarhaller in runtime harness
add 722a7e1 CR: [BEAM-3287] Use model pipelines in Go SDK
add 99f3997 CR: [BEAM-3287] Use model pipelines in Go SDK
add 92883b2 Merge pull request #4213: [BEAM-3287] Use model pipelines in
Go SDK
add 0d5d495 Fix name of syscallx ErrUnsupported
add 18a7647 Allow any named type to be registered and serialized as
external
add 4fa1d78 Add more package comments for core packages
add 2d695e7 Make Go SDK External a graph primitive
add 4a534b0 Merge pull request #4237: [BEAM-3294] Make Go SDK external
transform a primitive
add b258467 Cache Go runtime symbol lookups
add 61558b9 Merge pull request #4256: [BEAM-3324] Cache Go runtime symbol
lookups
add 9bdae2d Merge pull request #4271: merge master into go-sdk
add cb6b6ba Fix storagePath variable.
add a4e4cb3 Merge pull request #4279: Fix storagePath variable.
add 14eb2cd [BEAM-3356] Add Go SDK int and varint custom coders (#4276)
add 610db85 BEAM-3361 Increase Go gRPC message size
add e49cde3 Go SDK runtime revamp
add 60216f1 Merge pull request #4265 from herohde/runtime2
add 6557632 Add a few function call overhead benchmarks
add fc550df Add type-specialized emitters
add a9dc174 BEAM-3324 improve symtab memory usage
add d85a502 BEAM-3324 improve symtab memory usage
add 03f96c6 BEAM-3324 improve symtab memory usage
add 412de6f Store objects in pool so they can be reused.
add e6f3af2 Add builtin varint coder
add f68f05e Merge pull request #4350: Integrate master into go-sdk
add d4b3624 Type-specialize custom decoders and encoders in Go SDK runtime
add 7bdffd6 Type-specialize iterators and side input in the Go SDK
add 9be0836 Add warnings if Go runtime registrations are overwritten
add 0b43308 Add reusable element coders in Go SDK runtime
add 2a0ba98 Merge pull request #4336 from herohde/runtime5
add 1379ad7 BEAM-3368 fix translation for external
add b7168ce Updated translater to preserve payload and its URN.
add 92cbda3 Merge pull request #4284 from wcn3/model_external_support
add 0db4090 Initial version of type-specialized general reflective calls
add 5ff88b0 Add general-purpose untyped callers in Go SDK runtime
add 2dc8363 Use fast caller for filter transform predicate
add 0f3ec80 CR: Clarified comment on caller template
add acac64e Merge pull request #4373 from herohde/runtime6
add 599b746 Fix value encoding for Create
add 334180c Merge pull request #4397
add 1fe55ca BEAM-3473: Fix GroupByKey iterators to be initialized.
add 864b359 BEAM-3474 Include stacks in panic messages.
add 0af2503 BEAM-3299: Add source reporting support.
add a86c25e Remove GetId() call from under lock.
add efa1c20 Add additional comments about concurrency invariants.
add c5a3ce0 Add initialization of active plans map.
add c67de93 Renamed Go runtime Caller to Func and added name
add 09e98e4 Use reflectx.Func as the fundamental function representation
add 8a9d916 CR: fix DynFn comments
add e019f19 CR: fix comments
add 92e3bbe Avoid reflect.Value conversions in Go runtime
add 76cc0a6 Merge pull request #4452 from herohde/runtime8
add cebdbb7 Allow grpcx.Dial to support overrides.
add b9b191f Merge pull request #4456: BEAM-3505: Allow grpcx.Dial to
support overrides.
add 927e46c Use a typeswitch instead of reflect.Convert when encoding
strings or bytes.
add da0d0df Update coder.go
add 9aa8fe4 Merge pull request #4474 from lostluck/byteencode
add 0f01201 Replace reflective convert to direct convert.
add 8cbd440 Merge pull request #4509
add 5386431 Merge branch 'master' into go-sdk
add 010272b Merge master into go-sdk
add 1f33d31 Fix beam.Combine to combine globally
add f7e4e41 Merge pull request #4556: Fix beam.Combine to combine globally
add 6834287 Add optional function registration to Go SDK runtime
add 4b69c36 Merge pull request #4582: [BEAM-3597] Add optional function
registration to Go SDK runtime
add bd97498 fixup! Remove reflection from varint codecs
add 52a598d [BEAM-3388] Remove reflection from varint codecs
add 07e534b Changed core GBK to CoGBK
add 66ecd91 Add CoGBK support to direct runner and Join example
add 79fb7e1 [BEAM-3316] Translate bundle descriptors directly to
execution plans in Go SDK
add 1cf50dd Translate CoGBK into GBK for Dataflow and model pipeline
runners
add b7dce98 CR: [BEAM-3302] Support CoGBK in the Go SDK
add a328e44 Merge pull request #4540 [BEAM-3302] Support CoGBK in the Go
SDK
add 21bc8cf [BEAM-3579] Fix textio.Write
add 6e71095 CR: fix Go SDK textio.Write
add 8cb9500 Merge pull request #4624
add 0dcbb0b Improve rendering of DOT diagrams.
add 5cb4ac6 Dot rendering improvement
add 8e8d5fb Fixing filename.
add e2d7408 Merge pull request #4654: CoGBK fixup - rename file
add 48da05a Integrate master into go-sdk branch
add ed922cb Update Go SDK coder constants
add 5e6db92 Merge pull request #4682: [BEAM-3684] Integrate master to
go-sdk and fixup coder urns
add b7529d7 Merge branch 'master' into go-sdk
add d960b53 Merge branch 'master' into go-sdk
new 45ff873 Merge master into go-sdk
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
pom.xml | 5 +-
sdks/go/README.md | 98 +
sdks/go/cmd/specialize/main.go | 240 +
sdks/go/cmd/symtab/main.go | 101 +
sdks/go/container/Dockerfile | 30 +
sdks/go/container/boot.go | 127 +
sdks/go/{ => container}/pom.xml | 87 +-
sdks/go/data/haiku/old_pond.txt | 3 +
sdks/go/data/shakespeare/hamlet.txt | 6045 ++++++
sdks/go/data/shakespeare/kinglear-hashtag.txt | 5525 +++++
sdks/go/data/shakespeare/kinglear.txt | 5525 +++++
sdks/go/data/shakespeare/romeoandjuliet.txt | 4766 +++++
.../examples/complete/autocomplete/autocomplete.go | 71 +
sdks/go/examples/contains/contains.go | 99 +
sdks/go/examples/cookbook/combine/combine.go | 106 +
sdks/go/examples/cookbook/filter/filter.go | 106 +
sdks/go/examples/cookbook/join/join.go | 111 +
sdks/go/examples/cookbook/max/max.go | 88 +
sdks/go/examples/cookbook/tornadoes/tornadoes.go | 119 +
.../debugging_wordcount/debugging_wordcount.go | 163 +
sdks/go/examples/forest/forest.go | 81 +
sdks/go/examples/grades/grades.go | 105 +
.../minimal_wordcount/minimal_wordcount.go | 101 +
sdks/go/examples/pingpong/pingpong.go | 131 +
sdks/go/examples/wordcap/wordcap.go | 75 +
sdks/go/examples/wordcount/wordcount.go | 168 +
sdks/go/examples/yatzy/yatzy.go | 139 +
sdks/go/pkg/beam/coder.go | 180 +
.../pkg/beam/{util/grpcx/dial.go => coder_test.go} | 34 +-
sdks/go/pkg/beam/combine.go | 78 +
.../syscall_default.go => core/funcx/doc.go} | 20 +-
sdks/go/pkg/beam/core/funcx/fn.go | 427 +
sdks/go/pkg/beam/core/funcx/fn_test.go | 224 +
sdks/go/pkg/beam/core/funcx/output.go | 76 +
sdks/go/pkg/beam/core/funcx/output_test.go | 48 +
sdks/go/pkg/beam/core/funcx/sideinput.go | 100 +
sdks/go/pkg/beam/core/funcx/sideinput_test.go | 70 +
sdks/go/pkg/beam/core/funcx/signature.go | 212 +
sdks/go/pkg/beam/core/funcx/signature_test.go | 212 +
sdks/go/pkg/beam/core/graph/bind.go | 325 +
sdks/go/pkg/beam/core/graph/bind_test.go | 207 +
sdks/go/pkg/beam/core/graph/coder/coder.go | 298 +
sdks/go/pkg/beam/core/graph/coder/int.go | 71 +
sdks/go/pkg/beam/core/graph/coder/int_test.go | 88 +
.../grpcx/dial.go => core/graph/coder/time.go} | 31 +-
sdks/go/pkg/beam/core/graph/coder/varint.go | 98 +
sdks/go/pkg/beam/core/graph/coder/varint_test.go | 94 +
.../syscall_default.go => core/graph/doc.go} | 20 +-
sdks/go/pkg/beam/core/graph/edge.go | 397 +
sdks/go/pkg/beam/core/graph/fn.go | 304 +
sdks/go/pkg/beam/core/graph/graph.go | 126 +
sdks/go/pkg/beam/core/graph/graph_test.go | 68 +
sdks/go/pkg/beam/core/graph/node.go | 70 +
.../syscall_default.go => core/graph/scope.go} | 27 +-
sdks/go/pkg/beam/core/graph/window/window.go | 71 +
sdks/go/pkg/beam/core/runtime/coderx/int.go | 98 +
sdks/go/pkg/beam/core/runtime/coderx/varint.go | 139 +
.../go/pkg/beam/core/runtime/coderx/varint_test.go | 84 +
sdks/go/pkg/beam/core/runtime/exec/coder.go | 319 +
sdks/go/pkg/beam/core/runtime/exec/cogbk.go | 171 +
sdks/go/pkg/beam/core/runtime/exec/combine.go | 257 +
sdks/go/pkg/beam/core/runtime/exec/combine_test.go | 67 +
sdks/go/pkg/beam/core/runtime/exec/datasink.go | 90 +
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 203 +
sdks/go/pkg/beam/core/runtime/exec/decode.go | 102 +
.../runtime/exec/discard.go} | 41 +-
sdks/go/pkg/beam/core/runtime/exec/emit.go | 126 +
sdks/go/pkg/beam/core/runtime/exec/encode.go | 102 +
sdks/go/pkg/beam/core/runtime/exec/flatten.go | 75 +
sdks/go/pkg/beam/core/runtime/exec/flatten_test.go | 49 +
sdks/go/pkg/beam/core/runtime/exec/fn.go | 201 +
sdks/go/pkg/beam/core/runtime/exec/fn_test.go | 256 +
sdks/go/pkg/beam/core/runtime/exec/fullvalue.go | 145 +
.../pkg/beam/core/runtime/exec/fullvalue_test.go | 72 +
sdks/go/pkg/beam/core/runtime/exec/input.go | 188 +
sdks/go/pkg/beam/core/runtime/exec/multiplex.go | 62 +
.../pkg/beam/core/runtime/exec/multiplex_test.go | 55 +
.../beam/core/runtime/exec/optimized/callers.go | 632 +
.../beam/core/runtime/exec/optimized/callers.tmpl | 69 +
.../beam/core/runtime/exec/optimized/decoders.go | 2407 +++
.../beam/core/runtime/exec/optimized/decoders.tmpl | 146 +
.../beam/core/runtime/exec/optimized/emitters.go | 14215 +++++++++++++
.../beam/core/runtime/exec/optimized/emitters.tmpl | 112 +
.../beam/core/runtime/exec/optimized/encoders.go | 2299 ++
.../beam/core/runtime/exec/optimized/encoders.tmpl | 146 +
.../runtime/exec/optimized/gen.go} | 19 +-
.../pkg/beam/core/runtime/exec/optimized/inputs.go | 21272 +++++++++++++++++++
.../beam/core/runtime/exec/optimized/inputs.tmpl | 181 +
sdks/go/pkg/beam/core/runtime/exec/pardo.go | 193 +
sdks/go/pkg/beam/core/runtime/exec/pardo_test.go | 97 +
sdks/go/pkg/beam/core/runtime/exec/plan.go | 153 +
.../runtime/exec/status.go} | 21 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 433 +
sdks/go/pkg/beam/core/runtime/exec/unit.go | 75 +
sdks/go/pkg/beam/core/runtime/exec/unit_test.go | 111 +
sdks/go/pkg/beam/core/runtime/exec/util.go | 72 +
sdks/go/pkg/beam/core/runtime/graphx/coder.go | 415 +
sdks/go/pkg/beam/core/runtime/graphx/coder_test.go | 107 +
sdks/go/pkg/beam/core/runtime/graphx/cogbk.go | 108 +
.../runtime/graphx/doc.go} | 27 +-
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 962 +
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 402 +
.../pkg/beam/core/runtime/graphx/translate_test.go | 90 +
sdks/go/pkg/beam/core/runtime/graphx/tree.go | 121 +
sdks/go/pkg/beam/core/runtime/graphx/user.go | 110 +
.../runtime/graphx/v1/gen.go} | 14 +-
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 821 +
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto | 232 +
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 308 +
.../runtime/harness/gen.go} | 14 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 291 +
sdks/go/pkg/beam/core/runtime/harness/init/init.go | 116 +
sdks/go/pkg/beam/core/runtime/harness/logging.go | 151 +
sdks/go/pkg/beam/core/runtime/harness/session.go | 215 +
.../go/pkg/beam/core/runtime/harness/session.proto | 69 +
.../core/runtime/harness/session/session.pb.go | 444 +
sdks/go/pkg/beam/core/runtime/init.go | 44 +
sdks/go/pkg/beam/core/runtime/options.go | 91 +
.../grpcx/dial.go => core/runtime/options_test.go} | 44 +-
sdks/go/pkg/beam/core/runtime/symbols.go | 93 +
sdks/go/pkg/beam/core/runtime/types.go | 62 +
sdks/go/pkg/beam/core/runtime/types_test.go | 74 +
sdks/go/pkg/beam/core/typex/class.go | 174 +
sdks/go/pkg/beam/core/typex/class_test.go | 104 +
sdks/go/pkg/beam/core/typex/fulltype.go | 400 +
sdks/go/pkg/beam/core/typex/fulltype_test.go | 125 +
sdks/go/pkg/beam/core/typex/special.go | 74 +
sdks/go/pkg/beam/core/util/dot/dot.go | 130 +
.../syscall.go => core/util/ioutilx/read.go} | 29 +-
sdks/go/pkg/beam/core/util/protox/any.go | 101 +
sdks/go/pkg/beam/core/util/protox/any_test.go | 83 +
.../grpcx/dial.go => core/util/protox/base64.go} | 37 +-
.../util/protox/protox.go} | 20 +-
.../grpcx/dial.go => core/util/protox/query.go} | 34 +-
sdks/go/pkg/beam/core/util/reflectx/call.go | 120 +
sdks/go/pkg/beam/core/util/reflectx/calls.go | 1341 ++
sdks/go/pkg/beam/core/util/reflectx/calls.tmpl | 72 +
sdks/go/pkg/beam/core/util/reflectx/functions.go | 45 +
.../syscall.go => core/util/reflectx/json.go} | 21 +-
sdks/go/pkg/beam/core/util/reflectx/tags.go | 74 +
sdks/go/pkg/beam/core/util/reflectx/types.go | 111 +
sdks/go/pkg/beam/core/util/symtab/symtab.go | 127 +
sdks/go/pkg/beam/create.go | 108 +
.../beam/{util/grpcx/dial.go => create_test.go} | 38 +-
sdks/go/pkg/beam/doc.go | 67 +
sdks/go/pkg/beam/encoding.go | 148 +
sdks/go/pkg/beam/external.go | 64 +
sdks/go/pkg/beam/flatten.go | 72 +
sdks/go/pkg/beam/forward.go | 94 +
sdks/go/pkg/beam/gbk.go | 104 +
sdks/go/pkg/beam/impulse.go | 46 +
sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 241 +
.../dial.go => io/bigqueryio/bigquery_test.go} | 35 +-
sdks/go/pkg/beam/io/textio/filesystem.go | 49 +
sdks/go/pkg/beam/io/textio/gcs/gcs.go | 130 +
.../grpcx/dial.go => io/textio/local/local.go} | 44 +-
sdks/go/pkg/beam/io/textio/textio.go | 207 +
sdks/go/pkg/beam/log/log.go | 183 +
.../{util/syscallx/syscall.go => log/standard.go} | 26 +-
sdks/go/pkg/beam/option.go | 72 +
.../grpcx/dial.go => options/gcpopts/options.go} | 27 +-
sdks/go/pkg/beam/pardo.go | 314 +
sdks/go/pkg/beam/partition.go | 124 +
sdks/go/pkg/beam/partition_test.go | 155 +
sdks/go/pkg/beam/pcollection.go | 90 +
sdks/go/pkg/beam/pipeline.go | 87 +
sdks/go/pkg/beam/runner.go | 51 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 367 +
sdks/go/pkg/beam/runners/dataflow/messages.go | 174 +
sdks/go/pkg/beam/runners/dataflow/translate.go | 448 +
sdks/go/pkg/beam/runners/direct/buffer.go | 165 +
sdks/go/pkg/beam/runners/direct/direct.go | 274 +
sdks/go/pkg/beam/runners/direct/gbk.go | 135 +
sdks/go/pkg/beam/runners/direct/impulse.go | 61 +
sdks/go/pkg/beam/runners/dot/dot.go | 55 +
sdks/go/pkg/beam/runners/session/session.go | 337 +
sdks/go/pkg/beam/testing/passert/passert.go | 214 +
sdks/go/pkg/beam/testing/ptest/ptest.go | 59 +
.../dial.go => transforms/filter/distinct.go} | 39 +-
.../go/pkg/beam/transforms/filter/distinct_test.go | 70 +
sdks/go/pkg/beam/transforms/filter/filter.go | 95 +
sdks/go/pkg/beam/transforms/filter/filter_test.go | 90 +
.../syscall.go => transforms/stats/count.go} | 23 +-
sdks/go/pkg/beam/transforms/stats/count_test.go | 69 +
sdks/go/pkg/beam/transforms/stats/max.go | 44 +
sdks/go/pkg/beam/transforms/stats/max_switch.go | 138 +
.../stats/max_switch.tmpl} | 30 +-
sdks/go/pkg/beam/transforms/stats/max_test.go | 134 +
sdks/go/pkg/beam/transforms/stats/mean.go | 91 +
sdks/go/pkg/beam/transforms/stats/mean_test.go | 121 +
sdks/go/pkg/beam/transforms/stats/min.go | 44 +
sdks/go/pkg/beam/transforms/stats/min_switch.go | 138 +
.../stats/min_switch.tmpl} | 30 +-
sdks/go/pkg/beam/transforms/stats/min_test.go | 121 +
sdks/go/pkg/beam/transforms/stats/sum.go | 44 +
sdks/go/pkg/beam/transforms/stats/sum_switch.go | 102 +
.../stats/sum_switch.tmpl} | 27 +-
sdks/go/pkg/beam/transforms/stats/sum_test.go | 117 +
sdks/go/pkg/beam/transforms/stats/util.go | 48 +
sdks/go/pkg/beam/transforms/top/top.go | 180 +
sdks/go/pkg/beam/transforms/top/top_test.go | 126 +
sdks/go/pkg/beam/util.go | 117 +
sdks/go/pkg/beam/util/gcsx/gcs.go | 40 +
sdks/go/pkg/beam/util/grpcx/dial.go | 12 +-
sdks/go/pkg/beam/util/syscallx/syscall.go | 6 +-
sdks/go/pkg/beam/util/syscallx/syscall_default.go | 4 +-
sdks/go/pkg/beam/validate.go | 79 +
sdks/go/pkg/beam/x/beamx/run.go | 41 +
sdks/go/pkg/beam/x/debug/head.go | 68 +
sdks/go/pkg/beam/x/debug/print.go | 97 +
sdks/go/pom.xml | 8 +
211 files changed, 89285 insertions(+), 366 deletions(-)
create mode 100644 sdks/go/README.md
create mode 100644 sdks/go/cmd/specialize/main.go
create mode 100644 sdks/go/cmd/symtab/main.go
create mode 100644 sdks/go/container/Dockerfile
create mode 100644 sdks/go/container/boot.go
copy sdks/go/{ => container}/pom.xml (69%)
create mode 100644 sdks/go/data/haiku/old_pond.txt
create mode 100644 sdks/go/data/shakespeare/hamlet.txt
create mode 100644 sdks/go/data/shakespeare/kinglear-hashtag.txt
create mode 100644 sdks/go/data/shakespeare/kinglear.txt
create mode 100644 sdks/go/data/shakespeare/romeoandjuliet.txt
create mode 100644 sdks/go/examples/complete/autocomplete/autocomplete.go
create mode 100644 sdks/go/examples/contains/contains.go
create mode 100644 sdks/go/examples/cookbook/combine/combine.go
create mode 100644 sdks/go/examples/cookbook/filter/filter.go
create mode 100644 sdks/go/examples/cookbook/join/join.go
create mode 100644 sdks/go/examples/cookbook/max/max.go
create mode 100644 sdks/go/examples/cookbook/tornadoes/tornadoes.go
create mode 100644 sdks/go/examples/debugging_wordcount/debugging_wordcount.go
create mode 100644 sdks/go/examples/forest/forest.go
create mode 100644 sdks/go/examples/grades/grades.go
create mode 100644 sdks/go/examples/minimal_wordcount/minimal_wordcount.go
create mode 100644 sdks/go/examples/pingpong/pingpong.go
create mode 100644 sdks/go/examples/wordcap/wordcap.go
create mode 100644 sdks/go/examples/wordcount/wordcount.go
create mode 100644 sdks/go/examples/yatzy/yatzy.go
create mode 100644 sdks/go/pkg/beam/coder.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => coder_test.go} (58%)
create mode 100644 sdks/go/pkg/beam/combine.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go => core/funcx/doc.go}
(64%)
create mode 100644 sdks/go/pkg/beam/core/funcx/fn.go
create mode 100644 sdks/go/pkg/beam/core/funcx/fn_test.go
create mode 100644 sdks/go/pkg/beam/core/funcx/output.go
create mode 100644 sdks/go/pkg/beam/core/funcx/output_test.go
create mode 100644 sdks/go/pkg/beam/core/funcx/sideinput.go
create mode 100644 sdks/go/pkg/beam/core/funcx/sideinput_test.go
create mode 100644 sdks/go/pkg/beam/core/funcx/signature.go
create mode 100644 sdks/go/pkg/beam/core/funcx/signature_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/bind.go
create mode 100644 sdks/go/pkg/beam/core/graph/bind_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/coder.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/int.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/int_test.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => core/graph/coder/time.go} (50%)
create mode 100644 sdks/go/pkg/beam/core/graph/coder/varint.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/varint_test.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go => core/graph/doc.go}
(63%)
create mode 100644 sdks/go/pkg/beam/core/graph/edge.go
create mode 100644 sdks/go/pkg/beam/core/graph/fn.go
create mode 100644 sdks/go/pkg/beam/core/graph/graph.go
create mode 100644 sdks/go/pkg/beam/core/graph/graph_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/node.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/graph/scope.go} (60%)
create mode 100644 sdks/go/pkg/beam/core/graph/window/window.go
create mode 100644 sdks/go/pkg/beam/core/runtime/coderx/int.go
create mode 100644 sdks/go/pkg/beam/core/runtime/coderx/varint.go
create mode 100644 sdks/go/pkg/beam/core/runtime/coderx/varint_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/coder.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/cogbk.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/combine.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/combine_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/datasink.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/datasource.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/decode.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/runtime/exec/discard.go} (52%)
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/emit.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/encode.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/flatten.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/flatten_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fn.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fn_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fullvalue.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/fullvalue_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/input.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/multiplex.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/multiplex_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/callers.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/callers.tmpl
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/decoders.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/decoders.tmpl
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/emitters.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/emitters.tmpl
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/encoders.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/encoders.tmpl
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/runtime/exec/optimized/gen.go} (51%)
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/inputs.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/optimized/inputs.tmpl
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/pardo.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/pardo_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/plan.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/runtime/exec/status.go} (71%)
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/translate.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/unit.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/unit_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/util.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/coder.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/coder_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/cogbk.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/runtime/graphx/doc.go} (50%)
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/serialize.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/translate.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/translate_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/tree.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/user.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/runtime/graphx/v1/gen.go} (71%)
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/datamgr.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/runtime/harness/gen.go} (71%)
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/harness.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/init/init.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/logging.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/session.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/session.proto
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/session/session.pb.go
create mode 100644 sdks/go/pkg/beam/core/runtime/init.go
create mode 100644 sdks/go/pkg/beam/core/runtime/options.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => core/runtime/options_test.go}
(57%)
create mode 100644 sdks/go/pkg/beam/core/runtime/symbols.go
create mode 100644 sdks/go/pkg/beam/core/runtime/types.go
create mode 100644 sdks/go/pkg/beam/core/runtime/types_test.go
create mode 100644 sdks/go/pkg/beam/core/typex/class.go
create mode 100644 sdks/go/pkg/beam/core/typex/class_test.go
create mode 100644 sdks/go/pkg/beam/core/typex/fulltype.go
create mode 100644 sdks/go/pkg/beam/core/typex/fulltype_test.go
create mode 100644 sdks/go/pkg/beam/core/typex/special.go
create mode 100644 sdks/go/pkg/beam/core/util/dot/dot.go
copy sdks/go/pkg/beam/{util/syscallx/syscall.go => core/util/ioutilx/read.go}
(67%)
create mode 100644 sdks/go/pkg/beam/core/util/protox/any.go
create mode 100644 sdks/go/pkg/beam/core/util/protox/any_test.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => core/util/protox/base64.go} (52%)
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
core/util/protox/protox.go} (72%)
copy sdks/go/pkg/beam/{util/grpcx/dial.go => core/util/protox/query.go} (52%)
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/call.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/calls.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/calls.tmpl
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/functions.go
copy sdks/go/pkg/beam/{util/syscallx/syscall.go => core/util/reflectx/json.go}
(63%)
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/tags.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/types.go
create mode 100644 sdks/go/pkg/beam/core/util/symtab/symtab.go
create mode 100644 sdks/go/pkg/beam/create.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => create_test.go} (57%)
create mode 100644 sdks/go/pkg/beam/doc.go
create mode 100644 sdks/go/pkg/beam/encoding.go
create mode 100644 sdks/go/pkg/beam/external.go
create mode 100644 sdks/go/pkg/beam/flatten.go
create mode 100644 sdks/go/pkg/beam/forward.go
create mode 100644 sdks/go/pkg/beam/gbk.go
create mode 100644 sdks/go/pkg/beam/impulse.go
create mode 100644 sdks/go/pkg/beam/io/bigqueryio/bigquery.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => io/bigqueryio/bigquery_test.go}
(57%)
create mode 100644 sdks/go/pkg/beam/io/textio/filesystem.go
create mode 100644 sdks/go/pkg/beam/io/textio/gcs/gcs.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => io/textio/local/local.go} (51%)
create mode 100644 sdks/go/pkg/beam/io/textio/textio.go
create mode 100644 sdks/go/pkg/beam/log/log.go
copy sdks/go/pkg/beam/{util/syscallx/syscall.go => log/standard.go} (61%)
create mode 100644 sdks/go/pkg/beam/option.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => options/gcpopts/options.go} (60%)
create mode 100644 sdks/go/pkg/beam/pardo.go
create mode 100644 sdks/go/pkg/beam/partition.go
create mode 100644 sdks/go/pkg/beam/partition_test.go
create mode 100644 sdks/go/pkg/beam/pcollection.go
create mode 100644 sdks/go/pkg/beam/pipeline.go
create mode 100644 sdks/go/pkg/beam/runner.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflow.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/messages.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/translate.go
create mode 100644 sdks/go/pkg/beam/runners/direct/buffer.go
create mode 100644 sdks/go/pkg/beam/runners/direct/direct.go
create mode 100644 sdks/go/pkg/beam/runners/direct/gbk.go
create mode 100644 sdks/go/pkg/beam/runners/direct/impulse.go
create mode 100644 sdks/go/pkg/beam/runners/dot/dot.go
create mode 100644 sdks/go/pkg/beam/runners/session/session.go
create mode 100644 sdks/go/pkg/beam/testing/passert/passert.go
create mode 100644 sdks/go/pkg/beam/testing/ptest/ptest.go
copy sdks/go/pkg/beam/{util/grpcx/dial.go => transforms/filter/distinct.go}
(56%)
create mode 100644 sdks/go/pkg/beam/transforms/filter/distinct_test.go
create mode 100644 sdks/go/pkg/beam/transforms/filter/filter.go
create mode 100644 sdks/go/pkg/beam/transforms/filter/filter_test.go
copy sdks/go/pkg/beam/{util/syscallx/syscall.go => transforms/stats/count.go}
(63%)
create mode 100644 sdks/go/pkg/beam/transforms/stats/count_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/max.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/max_switch.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
transforms/stats/max_switch.tmpl} (68%)
create mode 100644 sdks/go/pkg/beam/transforms/stats/max_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/mean.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/mean_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/min.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/min_switch.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
transforms/stats/min_switch.tmpl} (68%)
create mode 100644 sdks/go/pkg/beam/transforms/stats/min_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/sum.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/sum_switch.go
copy sdks/go/pkg/beam/{util/syscallx/syscall_default.go =>
transforms/stats/sum_switch.tmpl} (70%)
create mode 100644 sdks/go/pkg/beam/transforms/stats/sum_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/util.go
create mode 100644 sdks/go/pkg/beam/transforms/top/top.go
create mode 100644 sdks/go/pkg/beam/transforms/top/top_test.go
create mode 100644 sdks/go/pkg/beam/util.go
create mode 100644 sdks/go/pkg/beam/validate.go
create mode 100644 sdks/go/pkg/beam/x/beamx/run.go
create mode 100644 sdks/go/pkg/beam/x/debug/head.go
create mode 100644 sdks/go/pkg/beam/x/debug/print.go
--
To stop receiving notification emails like this one, please contact
[email protected].
