See <https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/267/display/redirect?page=changes>
Changes: [jamalone] Initial add [stefano] Fix duplicated Stephan Ewen [stefano] Fix broken mailto [jamalone] Fix for timezone [jamalone] Addition of Twitter and Blog details to index. [klk] Add apache ID 'kenn' for Kenneth Knowles [jamalone] New blog post; changes to logos. [jamalone] Fix for blog post. [jamalone] Edit to index page [Max] change output directory to "content" [jbonofre] Fix jbonofre e-mail [mxm] Team Updates [takidau] Capability matrix page + blog post: - Content as discussed in this [davor] Add content files missed in PR2 due to an overly aggressive .gitignore [takidau] Blog page "read more" links + README.md for the site. [takidau] Remove trailing slash in blog "read more" links [mrtn] Fix 'getting started page' link in homepage alert message [davor] Add apache_ids for everyone that has one at this time. [davor] Add contribution guide [takidau] Capability matrix updates: - Add sliding window support for batch Spark [davor] Addition of public materials page/post [jamalone] Addition of public meetings page [jamalone] Update for second meeting [jamalone] Update for email list [jamalone] Update to meeting page [jamalone] Addition of material page; nav fixes [ddonnelly] Added the /docs/ directory and a landing page at docs/index.md. Added [iemejia] Add button to make the navigation bar accesible in mobile browsing [jamalone] Addition of PCollection blog post; blog fix [jamalone] Addition of sAF blog; delay of PCollection post. [mxm] [BEAM-103] update capability matrix [jamalone] Update date for blog post [robertwb] Remove duplicate paragraph. [robertwb] Fix blog post typo. [davor] Add Flink Batch Runner Blog Post [davor] Add a blog post about the first release [davor] Add a page about releases [davor] Add Javadoc for version 0.1.0-incubating [jamalone] Fixed HTML errors; added link and HTML tests via rake [github] added incubation disclaimer [github] fixed typo [fjp] Merged in pullhttps://github.com/apache/beam-site/pull/25 and [fjp] Adding Travis CI YAML config [dhalperi] [BEAM-500] Finished skeleton of new website layout [dhalperi] [BEAM-510] releases: use ASF download script to balance load on Apache [fjp] Fixed remaining rake failure. [klk] Moved javadoc under /learn/sdks/java and added the links back [fjp] Addressed BEAM-514 (add required asf links) [fjp] Addressed Beam-515 (apache logos) [dhalperi] Minor fixups [dhalperi] Navbar: move ASF dropdown to right side [dhalperi] Minor style tweaks for alignment [dhalperi] Fix ASF header collapsing [dhalperi] Added a half birthday blog post, as discussed on dev@. [dhalperi] Update to today [dhalperi] Add a link to the Apache Code of Conduct [klk] Revise Beam programming guide for new DoFn [jasonkuster] Add testing guide to website. [mxm] [BEAM-102] update capability matrix [jason] Updated per PR comments [fjp] Added toc [dhalperi] Update downloads link to 0.2.0-incubating [dhalperi] Upgrade Javadoc to 0.2.0-incubating [dhalperi] Build fix for Travis [terrencehan] minor: remove duplicate words [fjp] Added blog post about Strata+Hadoop World tutorial. Added Jesse Anderson [fjp] Regenerate html and make 'rake test' pass. [fjp] Added bundler gem to README. [fjp] Update mailing lists to use new lists site. [fjp] Add Design Principles (take from the original Beam technical vision [fjp] Remove page title from TOCs [fjp] Updated Contribution Guide & added Work In Progress * Added section on [fjp] Add Blog Post on TestStream [dhalperi] Add release guide to the website [fjp] Updated website instructions to use Bundler correctly for managing ruby [fjp] Turn on existing syntax highlighting styles. [fjp] Enable syntax highlighting in @tgroh's blog. [davor] Update to Beam committers [fjp] Removed TriggerTester, obsoleted in [fjp] Add IDE setup instructions [fjp] Reordered IDE section after code checkout and tweaked heading levels. [davor] [BEAM-501] New skin proposal [fjp] Add a first version of the WordCount walkthrough. [fjp] Temporarily disable tests for v2/, since the links are all fake [fjp] Add downstream merge exemption to contribution guide [swegner] Add IntelliJ Checkstyle documentation [aljoscha.krettek] Add pointers to 0.3.0-incubating release artifacts [davor] Flink compatibility matrix: drop redundant text [davor] [BEAM-752] Infrastructure for toggling code snippets in documentation [davor] Add Mobile gaming walkthrough [davor] New structure [davor] Move Javadoc from learn/sdks to documentation/sdks. [davor] [BEAM-890] Update compatibility matrix for Spark. [dhalperi] Add missing site.baseurls [dhalperi] Build in a separate content directory to catch bad links [davor] Move files [davor] Update README.md and _config.yml after moving the source to the src/ [fjp] Initial pass on a real Quickstart [davor] Remove unnecessary build timestamps in the generated site [davor] Fix typo in file name (materix -> matrix) [davor] BEAM-845 Update navigation and runner capability matrix to include Apex. [davor] [BEAM-902] Added runner toggles [BEAM-752] Infrastructure for toggling [davor] Introduce site.release_latest global variable to facilitate Javadoc [dhalperi] Add note about gpg-agent to release-guide [dhalperi] Ignore more generated/internal Javadoc in Release Guide [dhalperi] Tweak gpg-agent text, move to GPG setup instructions [davor] Add tool to fix links. [jason] Remove parser since it's not on the executors. [davor] Update beam_team Committer IDs [thw] Update Apex runner info after merge to master. [github] "before" instead of "behind" for "early" elements [davor] [BEAM-505] Fill in the documentation/runners/direct portion of the [davor] [BEAM-508] Fill in the documentation/runners/dataflow portion of the [davor] Use jekyll site.release_latest variable in direct runner sample pom.xml [davor] Add Jenkins job definitions for existing projects using Jenkins DSL [davor] Update content on v2 landing page [fjp] Add Python SDK page and add it to /documentation/index.md [fjp] Rename /get-started/releases to get-started/downloads. Delete old [aljoscha.krettek] [BEAM-899] Add Flink Instructions to quickstart.md [davor] Remove Apex runner from work in progress [aljoscha.krettek] [BEAM-1007] Fixed toggles code, to work in safari [davor] Update download page to specify backward-compatibility guarantees [davor] Quickstart: suggest using nightly build temporarily [davor] Add maturity model assessment [davor] Add commiter/PMC bar information [davor] Add contribution diversity [davor] Update documentation/pipelines/design-your-pipeline.md. Add 5 images for [davor] Add external link icon to Java SDK API reference, and change to open in [davor] Update main landing page and Beam Overview page [davor] [BEAM-277] Add transforms section to the programming guide [dhalperi] [BEAM-506] Fill in the documentation/runners/flink portion of the [dhalperi] Website: minor typographical corrections [davor] Simplify Flink Runner instructions for running on cluster [davor] fixed command to run WordCount example on direct runner, now in the site [davor] Update Additional Resources page [davor] [BEAM-507] Fill in the documentation/runners/spark portion of the [klk] Rename PTransform apply to expand throughout [tgroh] [BEAM-835] Added IntelliJ instructions to the contribution guide [tgroh] Reword IntelliJ Annotation Processor section [davor] [BEAM-900] Spark quickstart instructions. [davor] Dataflow quickstart fix [davor] Add DOAP file for the project [davor] Remove incubating references [davor] [BEAM-1220] Update team.md with post-graduation committer list [davor] [BEAM-1163] Add signature keys to the vote template (release guide) [davor] Add clarification to original pull request [davor] Update website pull request instructions in the Release Guide [davor] Additional clarifications to pull request [davor] Update Release Guide: exclude apex runner translation package from the [davor] Updated Wordcount, Mobile gaming, and Using the Docs pages. [dhalperi] Added python supports in Programming Guide [davor] Update Release Guide as part of the post graduation tasks [dhalperi] Rename apache_beam.utils.options to apache_beam.utils.pipeline_options [davor] Add .htaccess to automatically forward HTTP to HTTP [jbonofre] Add 0.4.0 release [dhalperi] test-your-pipeline: use site.baseurl, site.release_latest, canonical [davor] Added post for 0.4.0 release with Apex runner addition. [davor] Publish a blog about Apache Apex runner [davor] Publish a blog about Apache Apex, second attempt [davor] testing.md: fix links to project [davor] Pass over the contribution guide [github] Limit the number of blog posts listed on the main page [github] Update copyright on each page [davor] Update maturity model to reflect graduation [davor] Blog post on Beam graduation [robertwb] Add plugin for snippet extraction. [robertwb] Replace literal code samples with extracted, tested snippets. [dhalperi] [BEAM-278] Add I/O section to programming guide [dhalperi] Update according to reviewer comments [dhalperi] minor fixes [davor] Clarify maturity of various runners. [jbonofre] Use PMC instead of PPMC [davor] Adds PTransform style guide [davor] WIP: Add Python to WordCount documentation [davor] Added Python Quickstart [davor] Update documentation to remove python-sdk branch references [altay] Add new committers to the team. [davor] Cleanup of links: relative instead of absolute, https instead of http, [davor] Blog: "Media recap of the Apache Beam graduation" [davor] Disable Travis coverage [davor] Standardize Jenkins suites into .jenkins directory [github] Fix typo in programming-guide.md [davor] Delete jobs due to move to main repo. [altay] Add page about python type safety and type hints [davor] Add 0.5.0 release [klk] Update the capability matrix for state and timers [fjp] Update programming guide with Python snippets and data encoding section. [davor] Add documentation to build the site with a jekyll container [melissapa] Add back composite transforms anchor to fix broken links [dhalperi] processBatch() is now called processBundle() [altay] Add new Managing Python Pipeline Dependencies page [altay] Update ParDo documentation for Python [davor] Add klk as a blog author [davor] Add blog post "Stateful Processing with Apache Beam" [klk] Fix typo in stateful processing post [iemejia] Fix some typos and small formatting issues. [klk] Touch up some punctuation in state blog post [davor] Clarify state further in capability matrix [altay] Add custom sources and sinks for Python page [dhalperi] [BEAM-475] update release instructions for javadoc [dhalperi] Minor tweaks [melissapa] Fix broken links due to code path changes [davor] Add Apache in front of initial Beam reference [sourabhbajaj] [BEAM-1320] Update release guide to create pydocs [sourabhbajaj] Fix broken links [staslevin] Adding testing related info: TestPipeline, API surface. [altay] Fix the quickstart documentation [altay] Update quickstart to have runner specific examples [thomas.haines] BEAM-1559 add HBase docs [davor] Add contrib guide comments about using squash & force push. [davor] Comment Ostatic link as it has been broken for a few weeks [iemejia] [BEAM-325] Add Slack details to website [mairbek] Typo fix in "Reading Data Into Your Pipeline" example. [wnsdud1861] Fixed typos (from distributed to distributed) [altay] Finding a typo in a code block [davor] Fix possible erranous naming conventions reference [azur] Add IntelliJ files to .gitignore [davor] [BEAM-1652] Code style instructions for IntelliJ in contribution guide. [davor] Documentation changes for 0.6.0 release. [sourabhbajaj] Add htaccess back [altay] Changes to web site related to recently released Python SDK [devon.meunier] Typo fix [davor] Blog post for 0.6.0 release with python sdk [kirpichov] Add jkff as committer [iemejia] Add iemejia as committer [davor] Add Pipeline I/O section to website - outline + move some existing [davor] Add chamikara as a committer [azur] Add aviemzur as committer [iemejia] Improve committing website changes section in the contribution guide [iemejia] Fix timezone to America/Los_Angeles [peihe0] Change peihe company affiliation to Alibaba. [davor] Simplify merge process section [davor] Add windowing section to programming guide [altay] Simplify python quick start. [davor] add code snippet to page design-your-pipeline [klk] Reword RunnableOnService to ValidatesRunner [aljoscha.krettek] Update Capability Matrix: Flink supports state/timers in batch mode [sourabhbajaj] Fix the link to the apache ICLA [sourabhbajaj] [BEAM-1711] Add information about extra requires in Python Quickstart [altay] Docker server command should run non-incremental build [altay] Add "In-Progress" info to the Built-in I/O Transforms page [altay] Add Read Transform content for Authoring I/O Transforms - Overview [sisk] Add to in-progress IO list and alphabetize [azur] Add JSON as a planned IO to built-in IOs page. [github] Update quickstart-java.md [dhalperi] [BEAM-1949] Associated doc changes for renaming sideOutput to output [dhalperi] add CoGroupByKey to chapter 'Using GroupByKey' [aljoscha.krettek] remove extra streaming refs in capability matrix: Flink state/timers [davor] Track RestIO in In-Progress I/O Transforms [dhalperi] [BEAM-1272] Align the naming of "generateInitialSplits" and [davor] [BEAM-1949] Doc changes for Python SideOutputValue to OutputValue rename [davor] [BEAM-1741] Update direct and Cloud Dataflow runner pages with Python [davor] Update overview page. [davor] update release notes post 0.6.0 release [aljoscha.krettek] Update flink page ref from language-python to language-py and add toggle [davor] [BEAM-919] Remove remaining old use/learn links from website src [davor] [BEAM-825] Fill in the documentation/runners/apex portion of the website [davor] Fix Apex runner instructions (pending review comments and other changes) [davor] [BEAM-1452] Add composite transforms section to programming guide [davor] [BEAM-1011] Add triggers content to the programming guide [altay] [BEAM-1695] Improve Python-SDK's programming guide [lcwik] Fix broken HDFS link on built-in IO page [altay] [BEAM-2142] Add composite transforms Python snippets to programming [altay] Add Python snippets to Mobile Gaming. [klk] [BEAM-969] Add a gearpump runner web page [klk] Minor touch-ups to Gearpump listing on overview [davor] add SQL_DSL to page 'work-in-progress' [davor] Transfer some content from Create Your Pipeline to the Programming [davor] Add trademark symbol to the Downloads page [altay] Remove README.md references from the website [altay] Migrate hadoop inputformat to website [altay] Move extension README files to website [melissapa] Add quickstart step re: Python 2.7.x requirement [altay] fix typos in hadoop io documentation [davor] Release guide: fix instructions [davor] Revise WordCount Example Walkthrough [altay] Quick start should not default to java quickstart [kirpichov] Rewrites the section on Coders to not talk about them as a parsing [kirpichov] [BEAM-1353] Style Guide fixups [dhalperi] [BEAM-501] Large visual refresh of Apache Beam website homepage [dhalperi] Shorten quotes and fix a typo [dhalperi] [BEAM-501] fix quotes [dhalperi] Update latest version metadata information [dhalperi] Add version 2.0.0 to Downloads page [dhalperi] Python: fix links to PipelineOptions after move from utils to options [dhalperi] Update Java Quickstart to stop using snapshot repos [davor] Add a draft of the Beam FSR blog post [github] TikaIO JIRA number fix [github] Added Cloud Spanner Connector [hadarh] Add 'how to run' directions to WordCount for all runners. [dhalperi] remove note about JIRA [dhalperi] Changing Aggregators to Metrics in capability matrix [dhalperi] fixups [dhalperi] Do not use BeautifulSoup prettify, which is known to corrupt HTML [chamikara] Updates contribution guide to include development and testing [iemejia] Add HadoopInputFormatIO example to read from Hive's HCatalog [klk] Add support for "no-toggle" code snippets [klk] Add Beam Runner Guide [iemejia] Update IO with CassandraIO [iemejia] Add Hive (HCatalog) to the list of available I/O transforms [altay] Mention python setup.py sdist and link to its docs [altay] [BEAM-2437] Input path should be flexible in quickstart [iemejia] Add Amazon DynamoDB example using HadoopInputFormatIO [aljoscha.krettek] Update to more recent Flink version in example [aljoscha.krettek] Add note about Scala version when setting up Flink cluster [github] Remove references to Travis. [github] No longer working title [mergebot] Update build-in IO list with AMQP [mergebot] Remove AMQPIO from in-progress list [mergebot] Fix missing closing element [mergebot] Add JingsongLee as committer [mergebot] Restore index.html and fix order [thw] update apex runner properties example [github] Remove incorrect info from testing page. [github] Drop the word reviewer from the contribution guide [mariagh] Fix broken links [mariagh] Update to latest version [mariagh] Address review comments [mariagh] Address review comments [mergebot] Includes Splittable DoFn in the capability matrix [mergebot] [BEAM-2500] List Amazon S3 File System as a planned I/O. [mergebot] Port of Google doc [mergebot] IO Testing, unit tests: update after readthrough [mergebot] fixup! IO Testing, unit tests: update after readthrough [mergebot] fixup! fixup! IO Testing, unit tests: update after readthrough [mergebot] [BEAM-2638] Add Elasticsearch (v2 and upcoming v5) and MQTT to Built-in [mergebot] [BEAM-2126] Add JStorm runner to ongoing projects. [mergebot] Add initial text for I/O Integration Test documentation [mergebot] fixup! Add initial text for I/O Integration Test documentation [mergebot] Updated IO IT docs based on PR feedback [mergebot] Update JDBC IO IT docs based on PR feedback. [kirpichov] Update a dead link to Jenkins builds. [kirpichov] Add Apache Solr to the list of Built-In I/O Transforms [kirpichov] A couple of fixes to website [kirpichov] Expands the sections on Coders and Validation in Style Guide [mxm] Update company affiliation [azur] Update PMC [altay] Update PMC [mergebot] Add MapReduce and Tez runners to ongoing projects. [mergebot] Splittable DoFn blog post. [mergebot] Add Manu Zhang to committers list [mergebot] add a page for DSLs:SQL [mergebot] fix typo issues [mergebot] add intro of BeamSqlRow, trigger settings of aggregation/join, [mergebot] typo fix [mergebot] change page dsls/sql.md into a blog post [mergebot] split into 1 blog + 1 development doc. [mergebot] fix broken link [mergebot] update to latest API/usages [mergebot] remove post page move internal imple to the end of page [altay] Publish release 2.1.0 [altay] Fix quote [mergebot] [BEAM-2780] Fix Window sample code in programming guide [takidau] [BEAM-2800] Revamp sections 1 and 2 of DSL SQL docs: - Expand BeamRecord [takidau] Remove TODO [altay] Add tfrecord io to builtin io list [mingmxu] add SQL grammar in BNF [mingmxu] correct the syntax of ORDER_BY LIMIT [klk] Add blog post with timely processing [kirpichov] Adds guidance on how to test transform configuration/validation [altay] [rebase] add MingminXu to team.md [altay] Add James Xu(xumingming) as committer [altay] Add Reuven Lax to committers list. [kirpichov] Fixes SDF blog post after rename of Match to FileIO.match [altay] Document stale PR policy [huafengw] Update Gearpump runner related pages [melissapa] [BEAM-2510] Link to SDK pages from the quickstarts [mergebot] [BEAM-2869] Add dates to releases on downloads page [mergebot] [BEAM-1534] Add docker images documentation to the website [mergebot] [BEAM-2726] Fix typo in programming guide [mergebot] [BEAM-893] Update mobile gaming example [mergebot] Update with review feedback [mergebot] [BEAM-2839, BEAM-2838] Add MapReduce runner to Beam asf-site. [mergebot] fixup: address comments. [mergebot] fixup: address comments. [mergebot] fixup: address comments. [melissapa] [BEAM-667] Verify and update wordcount snippets [melissapa] Update with review feedback [melissapa] Update with review feedback [melissapa] [BEAM-2039] Number programming guide chapters [melissapa] Updates with review feedback [mergebot] Update Mapreduce capability matrix when/how entries [mergebot] Remove extraneous type variable from StateSpec in state blog post [melissapa] Add Cloud Resource Manager API to the required API list [melissapa] Add new execution model page and associated images [melissapa] Update with review feedback and new diagrams [melissapa] More updates from review feedback [basti.lj] [BEAM-2666] Add JStorm runner to Beam asf-site. [mergebot] Update source-repository.md [mergebot] [BEAM-2335] Document how to run NeedsRunner tests. [mergebot] fixup! [BEAM-2335] Document how to run NeedsRunner tests. [mergebot] Manually add missed files from content dir [iemejia] [BEAM-2727] Add build information to the vote email template [altay] [BEAM-2853] Move the nexmark documentation to the website [altay] fixups [melissapa] Fix dead link [mergebot] [MINOR] Updated hero and cards background to use background-size: cover [jbonofre] Add RedisIO in the built-in set [jbonofre] Move Elasticsearch v5.x from in-progress to built-in [altay] [BEAM-664] Update docs: MinimalWordCount in Java is intentionally [herohde] [BEAM-3121] Remove broken docker script and documentation [klk] New top menu plus side nav layout [mergebot] [BEAM-1934] Add more CoGroupByKey content/examples [mergebot] Update with Java snippet tags [mergebot] Add melap as committer [mergebot] Add page for the portability framework [mergebot] Add portability to contribute section [mergebot] Update PMC [mergebot] [BEAM-3123] Update status of BEAM-1035 and BEAM-1037 in Nexmark. [mergebot] Update IntelliJ Checkstyle instructions [mergebot] Address [mergebot] Address [pgerver] [BEAM-3162] Add IBM Streams Runner to compatability matrix [github] Update testing docs to account for Python. [echauchot] Add Etienne Chauchot as a committer [jbonofre] Update SCM URL to use gitbox instead of git-wip-us [echauchot] [BEAM-2853] Fix left menu in nexmark page [echauchot] [BEAM-2853] Move nexmark page to documentation/sdks/java/nexmark [melissapa] Fix ptransform guide links to use site variable [altay] Capabilities Matrix: Stateful processing support in Apex Runner [relax] documentation changes for 2.2.0-RC1 [relax] Cherry pick for pr/3578 [herohde] Add links to the Go SDK [mergebot] Add Reuven Lax as PMC member. [mergebot] Update release date for 2.2.0 release [melissapa] Fix broken SQL branch links [klk] Move reporter.apache.org step to finalize section [klk] Expand instructions for pushing release version tag [klk] Make contribution-guide.md markdown readable [klk] Revamp the contribution guide [mergebot] [BEAM-3338] Add SDK specific pages to SDK left nav [mergebot] [BEAM-3374] Fix missing/stretched images, improve alt text [mergebot] [BEAM-2664] Make latest stable version more visible [mergebot] Adding update to Wikipedia to release guide. [mergebot] Fixed an example of shading the application jar to it run on Spark [mergebot] Add link to Scio (Scala) interface. [mergebot] Manually set hight of scala logo. [iemejia] [BEAM-3467] Remove Java 7 from the docker development images [melissapa] Fix broken JIRA component links [mergebot] fix language tabs after a language specific paragraph [melissapa] [BEAM-3117] Fix reference to bundles and windows [jbonofre] Add 2017 look back blog post [melissapa] Fix broken links due to code directory moves [github] Replaces sources/sinks in programming guide with IOs [jbonofre] Announce 2.3.0 release [mergebot] Add blog post for the release of Beam 2.3.0 [aromanenko.dev] Fixed number of contributors and duplication of name of Alexey Romanenko [taro.murao] Java variable names should be in CamelCase [mergebot] Clarify runtime type checking documentation [mergebot] Add nightly snapshot build to the release guide [mergebot] [BEAM-3430] Update workdcount example for Java8 for Java SDK [mergebot] Add procedure to create a self executing Jar for DataFlow [mergebot] Don't pass the pom.xml as an input [mergebot] Updates list of built-in IO transforms [klk] Update Gemfile.lock [klk] Explicitly define section id due to kramdown id generation changes [altay] Update site to mention the 2 Python lint envs [lcwik] [BEAM-3846] Add a link to be able to self join the Apache Beam slack [lcwik] [BEAM-3852] Update slack channel link and slack invite for new channel. [robertwb] Update instructions on cryptographic hashes. [mergebot] Increase link checking timeouts [mergebot] Update SQL doc to match new APIs [mergebot] Update current release to 2.4.0. [mergebot] Update download page (including signature and checksum). [mergebot] Add instructions on how to build and use the Python tarball. [mergebot] Update nexmark support matrix for timer and state APIs [echauchot] Make minor fixes to Nexmark page [echauchot] Use {{ site.release_latest }} instead of [version] [echauchot] Make gs paths coherent for runs on dataflow runner [mergebot] expand and collpase navigation side bar new feature [mergebot] mark active item on navigation side bar [melissapa] [BEAM-4026] Remove link to deleted Go SDK branch [melissapa] parallelise test [mergebot] [BEAM-3252] Updating contributor's guide to reference Gradle. [mergebot] Fix project names to reflect changes required for release. [mergebot] Drop using gradle clean in favor of --rerun-tasks [mergebot] [BEAM-3252] Adding a step to the IntelliJ setup for gradle. [mergebot] [BEAM-3252] Removing Annotation Processing section. [mergebot] Stop recommending we use auto-import [mergebot] Update checkstyle comment to correspond to the current version [mergebot] Update instructions to run linter. [mergebot] Change first occurance of MyOptions to PipelineOptions [ehudm] Document Gradle command for testing Beam SDK. [melissapa] [BEAM-4101] Fix broken links [melissapa] Update Eclipse documentation for Gradle build. [melissapa] [BEAM-3036] Adding Checkstyle version warning. [melissapa] Add TODO with linked JIRA to update Docker documentation for Gradle. [robertwb] Ignore jstorm.io to unblock presubmits. [mergebot] Friendlier contribution guide; link to starter tasks [mergebot] Cleanup now that GitHub is not the mirror. [mergebot] Update contribution guide to reflect current practices. [mergebot] Further refinement. [mergebot] Update Nexmark launch instructions for Gradle [lukasz.gajowy] [BEAM-4119] Update i/o testing documentation to use gradle [mergebot] Add Jason Kuster as Committer [github] Adding pabloem to the list of team members. [github] Sorting by last name [github] Adding gris as committer as well. [mergebot] Document Java extensions for parsing Apache HTTPD logfiles and Useragent [mergebot] Fixed some layout issues [mergebot] Moved the 3rd party extensions to a separate page [mergebot] [BEAM-4177] Clarify thread contraint in Programming Guide 4.3.2 [coheigea] Typo fix in the programming guide [klk] Add "Become a Committer" with PMC guidelines [klk] Remove Team page in favor of link to ASF roster [klk] Remove/correct references to Team page [klk] Do not validate links from datatorrent.com [melissapa] Updates for Python streaming [klk] Add top-level Community section [klk] Remove obsolete Maturity Model section [klk] Move Release Guide to committers section [klk] Move committer section to the bottom (we can find it :-) [klk] Streamline the contribution section [klk] Google has scrapped research, now only AI [klk] Temporarily suppress validation of ai.google links [mergebot] Add link to portability support sheet [valentyn] Add a Python 3 entry to the list of ongoing projects. [mergebot] Add Go SDK quickstart and minimal documentation page [mergebot] Add Go to the Wordcount example [mergebot] Fix mailing list links and docker image [github] Update index.md [klk] Reorganize SQL documentation into sections [swegner] Remove DoFnTester from testing guide. [mergebot] Fix typo in becoming-a-committer docs [timrobertson100] [BEAM-4361] Document usage of HBase TableSnapshotInputFormat [mergebot] Add Apache Kudu IO to the planned IOs table [mergebot] Add next Java LTS to the WIP section of the contribution guide [mergebot] Add PubSubIO to Python IO table row [mergebot] Change ParquetIO status in built-in I/O section [lcwik] Add link to design docs and template to the contribution guide. [lukasz.gajowy] Add Performance Testing to "Work in Progress" section [marek.simunek] [BEAM-4368] add euphoria java 8 dsl documentation section [aromanenko.dev] Added design docs page [aromanenko.dev] Added first intro sentence. [aromanenko.dev] Moved one doc to other section and added several new docs [aromanenko.dev] Added more docs [mergebot] Add SQL area to ongoing projects [mergebot] [BEAM-4488] How to install Python SDK from source [BEAM-4489] Document [mergebot] More changes [mergebot] updates after review [mergebot] Updating release guide to use Gradle commands. [mergebot] Addressing comments. Improving commands. [mergebot] Addressing comments. Improving commands. [mergebot] Update release guide for publishing source zip from GitHub [mergebot] Addressing TODOs [mergebot] Address comments. [mergebot] Improving. [mergebot] Add more updates [mergebot] More updates [mergebot] Addressed Alan's comments [mergebot] Add instructions about how to stage python binaries [mergebot] copy renamed zi[ file to sdks/python/staging subdir [mergebot] Update gradle publish with --no-parallel --no-daemon [mergebot] Add Python Kafka to list of in-progress I/O transforms. [echauchot] Add Etienne Chauchot's ongoing work to the contribute page. [mergebot] [BEAM-2852] Update Nexmark documentation with Kafka support [mergebot] [BEAM-4479] Fixed document for Coder [swegner] Fix ErrorProne violations in fn harness and enable failOnWarnings [swegner] fixup! Fix ErrorProne violations in fn harness and enable failOnWarnings [swegner] Temporarily exclude website sources from RAT [swegner] fixup! Fix ErrorProne violations in fn harness and enable failOnWarnings [iemejia] [BEAM-4570] Use pagination in [ehudm] Jupyter notebook for monitoring precommit job times [matthias] Futurize examples subpackage [kirpichov] Propagates artifact retrieval token in Flink runner and to the Java [robertwb] Add Impulse primitive to Python SDK [robertwb] Expand Read to Impulse | ParDo(Split) | ParDo(Read) in Python SDK [apilloud] [SQL] Correctly represent pipeline [apilloud] [SQL] Cleanup table IO functions [apilloud] [SQL] Simplify overly verbose names [apilloud] [SQL] Add each Calcite node to Beam graph once [apilloud] [SQL] Seperate parse from pipeline construction [swegner] fixup! Fix ErrorProne violations in fn harness and enable failOnWarnings [lcwik] [BEAM-4592] Make Dataflow understand kind:varint as a well known since [kirpichov] A couple more trivial changes extracted from #5662 [lcwik] Update Dataflow container images [katarzyna.kucharczyk] Created static retry method which accepts retriable method, amount of [robertwb] Update impulse-based read. [kenn] Make BeamRelNode take a PCollectionList<Row> always [rober] Fix build break from PR 5676, which modified method signatures. [migryz] Implemented working prototype for PostCommit jobs [migryz] Update post commit jobs to utilize new helper methods [ankurgoenka] Fix cache references in portable flink runner [Pablo] Adding ignore parameters for gitattributes [swegner] Remove submodule [migryz] Fix typo in _ghprb suffix [migryz] Update suffix to be PullRequest [ccy] Revert #5689 to fix build [ankurgoenka] Remove unused import [github] Make suffix shorter [kirpichov] Make ImmutableExecutableStage constructors public [kirpichov] [BEAM-4285] Implement Flink batch side input handler ------------------------------------------ [...truncated 92.61 MB...] Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map as step s17 Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize as step s18 Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key as step s19 Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign as step s20 Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey as step s21 Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable as step s22 Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator addStep INFO: Adding WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map as step s23 Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: Staging pipeline description to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/ Jun 21, 2018 1:41:01 AM org.apache.beam.runners.dataflow.util.PackageUtil tryStagePackage INFO: Uploading <63502 bytes, hash 9XT9I2YZcIq2RxgHlozFmw> to gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/tmp/staging/pipeline-9XT9I2YZcIq2RxgHlozFmw.pb Dataflow SDK version: 2.6.0-SNAPSHOT Jun 21, 2018 1:41:02 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-06-20_18_41_01-13396469126744618791?project=apache-beam-testing Submitted job: 2018-06-20_18_41_01-13396469126744618791 Jun 21, 2018 1:41:02 AM org.apache.beam.runners.dataflow.DataflowRunner run INFO: To cancel the job using the 'gcloud' tool, run: > gcloud dataflow jobs --project=apache-beam-testing cancel > --region=us-central1 2018-06-20_18_41_01-13396469126744618791 Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:01.991Z: Autoscaling is enabled for job 2018-06-20_18_41_01-13396469126744618791. The number of workers will be between 1 and 1000. Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:02.017Z: Autoscaling was automatically enabled for job 2018-06-20_18_41_01-13396469126744618791. Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:04.502Z: Checking required Cloud APIs are enabled. Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:04.733Z: Checking permissions granted to controller Service Account. Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:07.759Z: Worker configuration: n1-standard-1 in us-central1-b. Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.199Z: Expanding CoGroupByKey operations into optimizable parts. Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.289Z: Expanding GroupByKey operations into optimizable parts. Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.317Z: Lifting ValueCombiningMappingFns into MergeBucketsMappingFns Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.479Z: Fusing adjacent ParDo, Read, Write, and Flatten operations Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.512Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource) Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.544Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.576Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.609Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.646Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.684Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.716Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow into WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.740Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous) Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.765Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize into WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.798Z: Fusing consumer WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key into WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.830Z: Unzipping flatten s12 for input s11.org.apache.beam.sdk.values.PCollection.<init>:329#20ff67585e33a8f6 Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.861Z: Fusing unzipped copy of WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow), through flatten WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/Flatten.PCollections, into producer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.884Z: Fusing consumer WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.915Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.947Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.969Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:08.994Z: Fusing consumer WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign into MapElements/Map Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.026Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.047Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.073Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial into WordCount.CountWords/Count.PerElement/Init/Map Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.095Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.116Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles into WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.128Z: Fusing consumer WordCount.CountWords/Count.PerElement/Init/Map into WordCount.CountWords/ParDo(ExtractWords) Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.159Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.181Z: Fusing consumer MapElements/Map into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.202Z: Fusing consumer WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify into WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.223Z: Fusing consumer WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify into WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.248Z: Fusing consumer WordCount.CountWords/ParDo(ExtractWords) into ReadLines/Read Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.564Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Create Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.589Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Create Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.612Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Create Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.621Z: Starting 1 workers in us-central1-b... Jun 21, 2018 1:41:12 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:09.778Z: Executing operation ReadLines/Read+WordCount.CountWords/ParDo(ExtractWords)+WordCount.CountWords/Count.PerElement/Init/Map+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Partial+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Reify+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Write Jun 21, 2018 1:41:18 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:17.936Z: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). Jun 21, 2018 1:41:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:28.502Z: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). Jun 21, 2018 1:41:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:28.535Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Jun 21, 2018 1:41:48 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:41:46.796Z: Workers have started successfully. Jun 21, 2018 1:43:06 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:06.829Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close Jun 21, 2018 1:43:08 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:06.883Z: Executing operation WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write Jun 21, 2018 1:43:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:19.558Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Close Jun 21, 2018 1:43:20 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:19.615Z: Executing operation WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Read+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/GroupByWindow+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnwritten+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/DropShardNum+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow) Jun 21, 2018 1:43:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:23.715Z: Executing operation s12-u40 Jun 21, 2018 1:43:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:23.896Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/CreateDataflowView Jun 21, 2018 1:43:24 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:24.040Z: Executing operation WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Create.Values/Read(CreateSource)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Reify.ReifyView/ParDo(Anonymous)+WriteCounts/WriteFiles/GatherTempFileResults/Reify.ReifyViewInGlobalWindow/Values/Values/Map+WriteCounts/WriteFiles/FinalizeTempFileBundles/Finalize+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Pair with random key+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/Window.Into()/Window.Assign+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Reify+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Write Jun 21, 2018 1:43:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:26.057Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Close Jun 21, 2018 1:43:27 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:26.115Z: Executing operation WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/Read+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/GroupByKey/GroupByWindow+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Reshuffle/ExpandIterable+WriteCounts/WriteFiles/FinalizeTempFileBundles/Reshuffle.ViaRandomKey/Values/Values/Map Jun 21, 2018 1:43:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:27.473Z: Cleaning up. Jun 21, 2018 1:43:29 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:43:27.547Z: Stopping worker pool... Jun 21, 2018 1:45:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:45:06.998Z: Autoscaling: Resized worker pool from 1 to 0. Jun 21, 2018 1:45:07 AM org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process INFO: 2018-06-21T01:45:07.191Z: Autoscaling: Would further reduce the number of workers but reached the minimum number allowed for the job. Jun 21, 2018 1:45:15 AM org.apache.beam.runners.dataflow.DataflowPipelineJob waitUntilFinish INFO: Job 2018-06-20_18_41_01-13396469126744618791 finished with status DONE. gsutil cat gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* | grep Montague: Montague: 47 Verified Montague: 47 gsutil rm gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/count* Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00000-of-00003... / [1 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00001-of-00003... / [2 objects] Removing gs://temp-storage-for-release-validation-tests/nightly-snapshot-validation/counts-00002-of-00003... / [3 objects] Operation completed over 3 objects. [SUCCESS] :beam-runners-google-cloud-dataflow-java:runQuickstartJavaDataflow (Thread[Daemon worker,5,main]) completed. Took 5 mins 34.431 secs. > Task :beam-runners-flink_2.11:runQuickstartJavaFlinkLocal [ERROR] Failed to execute goal on project word-count-beam: Could not resolve dependencies for project org.example:word-count-beam:jar:0.1: Could not transfer artifact org.apache.beam:beam-sdks-java-core:jar:2.6.0-20180620.193404-15 from/to test.release (https://repository.apache.org/content/repositories/snapshots): GET request of: org/apache/beam/beam-sdks-java-core/2.6.0-SNAPSHOT/beam-sdks-java-core-2.6.0-20180620.193404-15.jar from test.release failed: Read timed out -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] Failed to execute goal on project word-count-beam: Could not resolve dependencies for project org.example:word-count-beam:jar:0.1: Could not transfer artifact org.apache.beam:beam-sdks-java-core:jar:2.6.0-20180620.193404-15 from/to test.release (https://repository.apache.org/content/repositories/snapshots): GET request of: org/apache/beam/beam-sdks-java-core/2.6.0-SNAPSHOT/beam-sdks-java-core-2.6.0-20180620.193404-15.jar from test.release failed: Read timed out -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] Failed command > Task :beam-runners-flink_2.11:runQuickstartJavaFlinkLocal FAILED :beam-runners-flink_2.11:runQuickstartJavaFlinkLocal (Thread[Task worker for ':' Thread 14,5,main]) completed. Took 31 mins 16.211 secs. FAILURE: Build completed with 2 failures. 1: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':beam-runners-direct-java:runMobileGamingJavaDirect'. > Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java'' > finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. ============================================================================== 2: Task failed with an exception. ----------- * What went wrong: Execution failed for task ':beam-runners-flink_2.11:runQuickstartJavaFlinkLocal'. > Process 'command '/usr/local/asfpackages/java/jdk1.8.0_152/bin/java'' > finished with non-zero exit value 1 * Try: Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output. Run with --scan to get full insights. ============================================================================== * Get more help at https://help.gradle.org BUILD FAILED in 31m 26s 8 actionable tasks: 7 executed, 1 from cache Publishing build scan... https://gradle.com/s/7y56wuudqze2q Build step 'Invoke Gradle script' changed build result to FAILURE Build step 'Invoke Gradle script' marked build as failure Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected] Not sending mail to unregistered user [email protected]
