See <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/3631/display/redirect?page=changes>
Changes: [herohde] Initial sketches of a Go SDK [herohde] Initial version of the direct style w/ direct runner. Incomplete. [herohde] Add Data as UserFn context w/ immediate value. [herohde] Added no-I/O wordcount for profiling. [herohde] Fleshed out possible approach to generic transformations. [herohde] Add “dag” example that use multiplexing and side input. [herohde] Added a more complex DAG example. [herohde] Add yatzy example with more complex construction-time setup [herohde] Add proto for Fn API [herohde] Add beam.Composite helper for the most common pattern to align with java [herohde] Move pipeline-construction time errors into an accumulator [herohde] Add Dataflow job and Fn API clients. Incomplete. [herohde] Add binary cross-compile and upload to Dataflow runner. Incomplete. [herohde] Add tentative runner indirection (default: local). [herohde] Made data flow runner detect user main for cross-compilation. [herohde] Remove error accumulation in favor of panic. [herohde] Improve Dataflow translation of coders, side input and composite names. [herohde] Fix name for AsView. [herohde] Use 2 grpc endpoints in harness [herohde] Add gRPC harness logging [herohde] Flesh out harness and serialization further. [herohde] Made the dataflow runner wait for job termination by default [herohde] beam: [herohde] beam: [herohde] combinefn.go: fix compilation issues [herohde] Improve dataflow serialization and execution. Incomplete. [herohde] Sleep 30 sec in wordcap to allow logs to propagate to Cloud Logging. [herohde] Move the 30s sleep for logging to the harness instead of in WordCap. [herohde] Post-review updates. [herohde] Doc updates. [herohde] Flesh out coders. Incomplete. [herohde] Added prototype implementation of more coders and the runner source. [herohde] dofn: illustrates how dofns are written. [herohde] beam: add viewfn and windowfn to side inputs match support Beam 1.0 [herohde] dofn: timers [herohde] Complete revamp: coders, graph and execution use element-wise [herohde] Fix coder encoding for Dataflow side input. Otherwise, the job is [herohde] Added more godoc comments to graph types. [herohde] Added more comments plus made local GBK use coder equality. [herohde] Added Flatten support and “forest” example that uses it. [herohde] Move bigqueryio to defunct [herohde] Make forest example print less [herohde] Add external struct registry and serialization. [herohde] Updated comments in node.go. [herohde] Replace real type with 'full type' since that's the current term. [herohde] Refactor Fn API dependency. [herohde] Added more comments to the runner/dataflow and runner/beamexec packages [herohde] Fix most go vet issues [herohde] Make core operations panic to cut down on the error propagation [herohde] Add more comments to the graph package. [herohde] Add DoFn wrapper to handle either function or (ptr to) struct [herohde] Fix remaining go vet warnings. [herohde] Code review for beam/graph/coder package. [herohde] Code review of the runtime/graphx package. [herohde] Remove Data options in favor of using a Fn struct [herohde] Code review of the beam/graph/userfn package. [herohde] Code review for beam/graph package. [herohde] godoc for runtime/graphx [herohde] Add support for []T and Combine functions [herohde] Add adapted documentation from the Java SDK to the beam package [herohde] Update snapshot of Fn API. [herohde] Add experiments flag to the Dataflow runner [herohde] Remove context arg from beamexec.Init [herohde] Migration to Runner API. [herohde] Add support for creating DOT graphs. [herohde] Make pretty printing of types and coders more concise [herohde] Add flexible Signature to aid type checking [herohde] Adding unit testability to harness translation. [herohde] Fix crash due to initialization order [herohde] Add CreateValues and Impulse [herohde] Add Runner API support for WindowingStrategy. [herohde] Run goimports on baseline. [herohde] Fix encoding of global window strategy. [herohde] Ensure the windowed value is atomically encoded. [herohde] Limit gRPC messages to max size. [herohde] Developer conveniences for running jobs. [herohde] Fix sends to not close the network channel. [herohde] Add re-iterable side input [herohde] Add per-key Combine [herohde] Add Min [herohde] Reorganize non-user-facing code into core [herohde] Make type register reject unnamed or predeclared types [herohde] Add type specialization tool [herohde] Don't run grpc plugin in generate phase. [herohde] Fix import reference path for runner API proto. [herohde] Revamp runner registration as _ imports [herohde] Add stats.Max and Mean [herohde] Add global pipeline options [herohde] Unify global and per-key combiners [herohde] Add beam convenience wrapper for imports and runner selection [herohde] Add session recording and CPU profiling to harness. [herohde] Add ptest and passert for testing pipelines [herohde] Add GCS and glob support to textio [herohde] Add BigQuery IO and examples [herohde] Adds a session runner for testing. [herohde] Add Partition and dynamic functions [herohde] Adding example that returns 10 words that contain provided search [herohde] Remove duplicate LOG line [herohde] Enable Combine Fns in Dataflow runner by modifying translation. [herohde] Fixing type bug by dropping T and using underlying type of value in [herohde] Adding graph validation at build time. [herohde] Import the Fn API changes. [herohde] Simple changes to support new Fn API coder changes. [herohde] Update translator to work with new Fn API changes. [herohde] Use appropriate equality tests. [herohde] Fix test to not use path of package. [herohde] Renaming directory to match package name. [herohde] Fixing random nits in comments. [herohde] Modify build command to avoid bash. [herohde] Fixing selected golint issues. [herohde] Addressing import review comments. [herohde] Add coder specialization for bytes/strings. [herohde] Adding unit tests to stats. [herohde] Fixing typo. [herohde] Add beam.External [herohde] Fix grpc.Dial calls to block properly. [herohde] Creates a symtab verifier by running Sym2Addr and Addr2Sym in a binary. [herohde] Add spec field to help interpretation of payload. [herohde] Use beam.T alias for typex.T etc with Go 1.9 [herohde] Move shared GCP options to a separate package [herohde] Update portability protos [herohde] Remove old source/sink from beam package [herohde] Add context-aware logging for both pipeline-submission time and runtime [herohde] Fix coder inference for strings. [herohde] Improve tornadoes example [herohde] Fix beam.External to map arguments correctly. [herohde] Added comments to yatzy and forest [herohde] Add comments to tornadoes from the java counterpart [herohde] Rename Pipeline Composite to Scope [herohde] Add 3 progressive wordcount examples [herohde] Clarify comments in wordcount pipelines [herohde] Add apache 2.0 license to files [herohde] Updates to examples. [herohde] Adding more godoc for the main beam package. [herohde] Update to new proto structure [herohde] Split Combine and fields in to global and per-key variants [herohde] Refactor Flatten of a single []T into Explode [herohde] Rename local runner to direct runner [herohde] Fix argument index error in ParDo execution [herohde] Add Apache copyright header to files that need it. [herohde] Made debug.Head not just work per bundle [herohde] Impose a total ordering on Fn parameters. [herohde] Rename Dedup to Distinct for consistency with other SDKs [herohde] Add coder to model coder translation [herohde] Simplify harness coder translation [herohde] Split Pipeline into Pipeline and Scope [herohde] Relocate Go SDK code [herohde] Fix Go SDK maven build [herohde] Move Go SKD to latest version of bigquery [herohde] Add Go SDK container image [herohde] Add Go SDK README [herohde] Update version for Go Dataflow pipelines [herohde] Make Scope a value type [herohde] Add Go graph/pipeline translation [herohde] Stage Go model pipeline for Dataflow [herohde] Use pipeline unmarhaller in runtime harness [herohde] CR: [BEAM-3287] Use model pipelines in Go SDK [herohde] CR: [BEAM-3287] Use model pipelines in Go SDK [herohde] Fix name of syscallx ErrUnsupported [herohde] Allow any named type to be registered and serialized as external [herohde] Add more package comments for core packages [herohde] Make Go SDK External a graph primitive [herohde] Cache Go runtime symbol lookups [wcn] Fix storagePath variable. [wcn] BEAM-3368 fix translation for external [robertwb] [BEAM-3356] Add Go SDK int and varint custom coders (#4276) [lcwik] BEAM-3361 Increase Go gRPC message size [herohde] Go SDK runtime revamp [lcwik] Add a few function call overhead benchmarks [lcwik] Add type-specialized emitters [lcwik] BEAM-3324 improve symtab memory usage [lcwik] BEAM-3324 improve symtab memory usage [lcwik] BEAM-3324 improve symtab memory usage [lcwik] Store objects in pool so they can be reused. [lcwik] Add builtin varint coder [herohde] Type-specialize custom decoders and encoders in Go SDK runtime [herohde] Type-specialize iterators and side input in the Go SDK [herohde] Add warnings if Go runtime registrations are overwritten [herohde] Add reusable element coders in Go SDK runtime [wcn] Updated translater to preserve payload and its URN. [herohde] Initial version of type-specialized general reflective calls [herohde] Add general-purpose untyped callers in Go SDK runtime [herohde] Use fast caller for filter transform predicate [herohde] CR: Clarified comment on caller template [herohde] Fix value encoding for Create [lcwik] BEAM-3473: Fix GroupByKey iterators to be initialized. [lcwik] BEAM-3474 Include stacks in panic messages. [lcwik] BEAM-3299: Add source reporting support. [lcwik] Remove GetId() call from under lock. [lcwik] Add additional comments about concurrency invariants. [lcwik] Add initialization of active plans map. [lcwik] Renamed Go runtime Caller to Func and added name [lcwik] Use reflectx.Func as the fundamental function representation [lcwik] CR: fix DynFn comments [lcwik] CR: fix comments [herohde] Avoid reflect.Value conversions in Go runtime [wcn] Allow grpcx.Dial to support overrides. [rober] Use a typeswitch instead of reflect.Convert when encoding strings or [github] Update coder.go [robert] Replace reflective convert to direct convert. [rober] Fix beam.Combine to combine globally [herohde] Add optional function registration to Go SDK runtime [rober] fixup! Remove reflection from varint codecs [herohde] Changed core GBK to CoGBK [herohde] Add CoGBK support to direct runner and Join example [herohde] [BEAM-3316] Translate bundle descriptors directly to execution plans in [herohde] Translate CoGBK into GBK for Dataflow and model pipeline runners [herohde] CR: [BEAM-3302] Support CoGBK in the Go SDK [herohde] [BEAM-3579] Fix textio.Write [herohde] CR: fix Go SDK textio.Write [wcn] Fixing filename. [ehudm] Integration test for Python HDFS implementation. [wcn] Improve rendering of DOT diagrams. [herohde] Update Go SDK coder constants [tgroh] Make Impulse#create() visible [tgroh] Update QueryablePipeline Factory Method Name [tgroh] Add JavaReadViaImpulse to core-construction [amyrvold] Beam runner inventory, run as a cron job, not on each CL ------------------------------------------ Started by GitHub push by tgroh Started by GitHub push by tgroh Started by GitHub push by tgroh Started by GitHub push by tgroh Started by GitHub push by tgroh Started by GitHub push by tgroh [EnvInject] - Loading node environment variables. Building remotely on beam2 (beam) in workspace <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/ws/> Cloning the remote Git repository Cloning repository https://github.com/apache/beam.git > git init > <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/ws/src> > # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # > timeout=10 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10 Fetching upstream changes from https://github.com/apache/beam.git > git fetch --tags --progress https://github.com/apache/beam.git > +refs/heads/*:refs/remotes/origin/* > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/* > git rev-parse origin/master^{commit} # timeout=10 Checking out Revision e3e4e3ce31b55f8cc403aa2b376814a1130ff49c (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f e3e4e3ce31b55f8cc403aa2b376814a1130ff49c Commit message: "Update QueryablePipeline Factory Method Name" > git rev-list --no-walk 5b99fe2ea44e1a0d8a6f802ca757d1a1cfab91cc # timeout=10 Cleaning workspace > git rev-parse --verify HEAD # timeout=10 Resetting working tree > git reset --hard # timeout=10 > git clean -fdx # timeout=10 [EnvInject] - Executing scripts and injecting environment variables after the SCM step. [EnvInject] - Injecting as environment variables the properties content SPARK_LOCAL_IP=127.0.0.1 [EnvInject] - Variables injected successfully. Parsing POMs Downloaded artifact http://repo.maven.apache.org/maven2/org/apache/apache/19/apache-19.pom ERROR: Failed to parse POMs java.io.IOException: remote file operation failed: <https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/ws/> at hudson.remoting.Channel@7574707c:beam2: hudson.remoting.ProxyException: hudson.maven.MavenModuleSetBuild$MavenExecutionException: org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM: Could not find artifact org.apache.beam:beam-sdks-go:pom:2.3.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 21, column 11 at hudson.FilePath.act(FilePath.java:993) at hudson.FilePath.act(FilePath.java:975) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:986) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:691) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1724) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: hudson.remoting.ProxyException: hudson.maven.MavenModuleSetBuild$MavenExecutionException: org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM: Could not find artifact org.apache.beam:beam-sdks-go:pom:2.3.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 21, column 11 at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1391) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1125) at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2760) at hudson.remoting.UserRequest.perform(UserRequest.java:207) at hudson.remoting.UserRequest.perform(UserRequest.java:53) at hudson.remoting.Request$2.run(Request.java:358) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to beam2 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693) at hudson.remoting.UserResponse.retrieve(UserRequest.java:310) at hudson.remoting.Channel.call(Channel.java:908) at hudson.FilePath.act(FilePath.java:986) at hudson.FilePath.act(FilePath.java:975) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:986) at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:691) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504) at hudson.model.Run.execute(Run.java:1724) at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:543) at hudson.model.ResourceController.execute(ResourceController.java:97) at hudson.model.Executor.run(Executor.java:429) Caused by: hudson.remoting.ProxyException: org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs: [FATAL] Non-resolvable parent POM: Could not find artifact org.apache.beam:beam-sdks-go:pom:2.3.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 21, column 11 at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:364) at hudson.maven.MavenEmbedder.buildProjects(MavenEmbedder.java:361) at hudson.maven.MavenEmbedder.readProjects(MavenEmbedder.java:331) at hudson.maven.MavenModuleSetBuild$PomParser.invoke(MavenModuleSetBuild.java:1328) ... 10 more Not sending mail to unregistered user hero...@google.com Not sending mail to unregistered user amyrv...@google.com Not sending mail to unregistered user git...@alasdairhodge.co.uk Not sending mail to unregistered user eh...@google.com Not sending mail to unregistered user ro...@frantil.com Not sending mail to unregistered user rob...@frantil.com Not sending mail to unregistered user w...@google.com
