Hi, I build a pipeline using RC 2.2 today and ran with runner on yarn. It worked seamlessly for unbounded sources. Couldn’t see any issues with my pipeline so far :)
Thanks,Nishu On Wed, 22 Nov 2017 at 18.57, Reuven Lax <[email protected]> wrote: > Who is validating Flink and Yarn? > > On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles <[email protected]> > wrote: > > > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov < > > [email protected]> wrote: > > > > > In the verification spreadsheet, I'm not sure I understand the > difference > > > between the "YARN" and "Standalone cluster/service". Which is Dataproc? > > It > > > definitely uses YARN, but it is also a standalone cluster/service. Does > > it > > > count for both? > > > > > > > No, it doesn't. A number of runners have their own non-YARN cluster > mode. I > > would expect that the launching experience might be different and the > > portable container management to differ. If they are identical, experts > in > > those systems should feel free to coalesce the rows. Conversely, as other > > platforms become supported, they could be added or not based on whether > > they are substantively different from a user experience or QA point of > > view. > > > > Kenn > > > > > > > Seems now we're missing just Apex and Flink cluster verifications. > > > > > > *though Spark runner took 6x longer to run UserScore, partially I guess > > > because it didn't do autoscaling (Dataflow runner ramped up to 5 > workers > > > whereas Spark runner used 2 workers). For some reason Spark runner > chose > > > not to split the 10GB input files into chunks. > > > > > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax <[email protected]> > > > wrote: > > > > > > > Done > > > > > > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw < > > > > [email protected]> wrote: > > > > > > > > > Thanks. You need to re-sign as well. > > > > > > > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax > > <[email protected] > > > > > > > > > wrote: > > > > > > FYI these generated files have been removed from the source > > > > distribution. > > > > > > > > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax <[email protected]> > > > wrote: > > > > > > > > > > > >> hmmm, I thought I removed those generated files from the zip > file > > > > before > > > > > >> sending this email. Let me check again. > > > > > >> > > > > > >> Reuven > > > > > >> > > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw < > > > > > >> [email protected]> wrote: > > > > > >> > > > > > >>> The source distribution contains a couple of files not on > github > > > > (e.g. > > > > > >>> folders that were added on master, Python generated files). The > > pom > > > > > >>> files differed only by missing -SNAPSHOT, other than that > > > presumably > > > > > >>> the source release should just be "wget > > > > > >>> https://github.com/apache/beam/archive/release-2.2.0.zip";? > > > > > >>> > > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml > > > > > >>> > > > > > >>> # OK? > > > > > >>> > > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES > > > > > >>> > > > > > >>> # Expected. > > > > > >>> > > > > > >>> Only in beam/: .git > > > > > >>> Only in beam/: .gitattributes > > > > > >>> Only in beam/: .gitignore > > > > > >>> > > > > > >>> # These folders are probably from switching around between > master > > > and > > > > > >>> git branches. > > > > > >>> > > > > > >>> Only in apache-beam-2.2.0: model > > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples > > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner > > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore > > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core > > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: sketching > > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: file-based-io-tests > > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs > > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > > maven-archetypes/examples/src/ > > > ma > > > > > >>> in/resources/archetype-resources: > > > > > >>> src > > > > > >>> Only in apache-beam-2.2.0/sdks/java/maven-archetypes/examples- > > > java8/ > > > > > >>> src/main/resources/archetype-resources: > > > > > >>> src > > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks > > > > > >>> > > > > > >>> # Here's the generated protos. > > > > > >>> > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_artifact_api_pb2_grpc.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_artifact_api_pb2.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_fn_api_pb2_grpc.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_fn_api_pb2.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_job_api_pb2_grpc.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_job_api_pb2.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_provision_api_pb2_grpc.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_provision_api_pb2.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_runner_api_pb2_grpc.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> beam_runner_api_pb2.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> endpoints_pb2_grpc.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> endpoints_pb2.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> standard_window_fns_pb2_grpc.py > > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > apache_beam/portability/api: > > > > > >>> standard_window_fns_pb2.py > > > > > >>> > > > > > >>> And some other sdist generated Python files. > > > > > >>> > > > > > >>> Only in apache-beam-2.2.0/sdks/python: .eggs > > > > > >>> Only in apache-beam-2.2.0/sdks/python: LICENSE > > > > > >>> Only in apache-beam-2.2.0/sdks/python: NOTICE > > > > > >>> Only in apache-beam-2.2.0/sdks/python: README.md > > > > > >>> > > > > > >>> Presumably we should just purge these files from the rc? > > > > > >>> > > > > > >>> > > > > > >>> FWIW, the Python tarball looks fine. > > > > > >>> > > > > > >>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov > > > > > >>> <[email protected]> wrote: > > > > > >>> > How can I specify a dependency on the staged RC? E.g. I'm > > trying > > > to > > > > > >>> > validate the quickstart per > > > > > >>> > https://beam.apache.org/get-started/quickstart-java/ and > > > > specifying > > > > > >>> version > > > > > >>> > 2.2.0 doesn't work I suppose because it's not released yet. > > > Should > > > > I > > > > > >>> pass > > > > > >>> > some command-line flag to mvn to make it fetch the version > from > > > the > > > > > >>> staging > > > > > >>> > area? > > > > > >>> > > > > > > >>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik > > > > <[email protected] > > > > > > > > > > > >>> > wrote: > > > > > >>> > > > > > > >>> >> Its open to all, its just that there are binding votes and > > > > > non-binding > > > > > >>> >> votes. > > > > > >>> >> > > > > > >>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev < > > > > > >>> >> [email protected]> wrote: > > > > > >>> >> > > > > > >>> >> > I have a process question: is the vote open for committers > > > only > > > > or > > > > > >>> for > > > > > >>> >> all > > > > > >>> >> > contributors? > > > > > >>> >> > > > > > > >>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik > > > > > >>> <[email protected]> > > > > > >>> >> > wrote: > > > > > >>> >> > > > > > > >>> >> > > +1, Approve the release > > > > > >>> >> > > > > > > > >>> >> > > I have verified the wordcount quickstart on the Apache > > Beam > > > > > website > > > > > >>> >> using > > > > > >>> >> > > Apex, DirectRunner, Flink & Spark on Linux. > > > > > >>> >> > > > > > > > >>> >> > > The Gearpump runner is yet to have a quickstart listed > on > > > our > > > > > >>> website. > > > > > >>> >> > > Adding the quickstart is already represented by this > > > existing > > > > > >>> issue: > > > > > >>> >> > > https://issues.apache.org/jira/browse/BEAM-2692 > > > > > >>> >> > > > > > > > >>> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn Tymofieiev < > > > > > >>> >> > > [email protected]> wrote: > > > > > >>> >> > > > > > > > >>> >> > > > I have verified: SHA & MD5 signatures of Python > > artifacts > > > in > > > > > >>> [2], and > > > > > >>> >> > > > checked Python side of the validation checklist on > > Linux. > > > > > >>> >> > > > > > > > > >>> >> > > > There is one known issue in UserScore example for > > Dataflow > > > > > >>> runner. > > > > > >>> >> The > > > > > >>> >> > > > issue has been fixed on master branch and does not > > > require a > > > > > >>> >> > cherry-pick > > > > > >>> >> > > at > > > > > >>> >> > > > this point. A workaround is to pass > --save_main_session > > > > > pipeline > > > > > >>> >> > option. > > > > > >>> >> > > > > > > > > >>> >> > > > RC4 looks good to me so far. > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > > >>> >> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth Knowles > > > > > >>> >> > <[email protected] > > > > > >>> >> > > > > > > > > >>> >> > > > wrote: > > > > > >>> >> > > > > > > > > >>> >> > > > > Hi all, > > > > > >>> >> > > > > > > > > > >>> >> > > > > Following up on past discussions and > > > > > >>> >> > > > > https://issues.apache.org/jira/browse/BEAM-1189 I > > have > > > > > >>> prepared a > > > > > >>> >> > > > > spreadsheet so we can sign up for validation steps > > that > > > > > must be > > > > > >>> >> done > > > > > >>> >> > > by a > > > > > >>> >> > > > > human. > > > > > >>> >> > > > > > > > > > >>> >> > > > > The spreadsheet for 2.2.0 is at > > > > > >>> >> > > > > https://s.apache.org/beam-2.2.0-release-validation. > > > > > Everyone > > > > > >>> can > > > > > >>> >> > edit, > > > > > >>> >> > > > so > > > > > >>> >> > > > > it is easy to sign up or add new rows. > > > > > >>> >> > > > > > > > > > >>> >> > > > > FYI the template is at > > > > > >>> >> https://s.apache.org/beam-release-validation. > > > > > >>> >> > > > After > > > > > >>> >> > > > > this release, I will update the rows in the template > > to > > > > > match > > > > > >>> >> > whatever > > > > > >>> >> > > we > > > > > >>> >> > > > > have added to the validation for 2.2.0. > > > > > >>> >> > > > > > > > > > >>> >> > > > > Kenn > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax > > > > > >>> >> > <[email protected] > > > > > >>> >> > > > > > > > > >>> >> > > > > wrote: > > > > > >>> >> > > > > > > > > > >>> >> > > > > > Hi everyone, > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > Please review and vote on the release candidate #4 > > for > > > > the > > > > > >>> >> version > > > > > >>> >> > > > 2.2.0, > > > > > >>> >> > > > > > as follows: > > > > > >>> >> > > > > > [ ] +1, Approve the release > > > > > >>> >> > > > > > [ ] -1, Do not approve the release (please > provide > > > > > specific > > > > > >>> >> > > comments) > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > The complete staging area is available for your > > > review, > > > > > which > > > > > >>> >> > > includes: > > > > > >>> >> > > > > > * JIRA release notes [1], > > > > > >>> >> > > > > > * the official Apache source release to be > > deployed > > > to > > > > > >>> >> > > > dist.apache.org > > > > > >>> >> > > > > > [2], > > > > > >>> >> > > > > > which is signed with the key with fingerprint > > B98B7708 > > > > > [3], > > > > > >>> >> > > > > > * all artifacts to be deployed to the Maven > > Central > > > > > >>> Repository > > > > > >>> >> > [4], > > > > > >>> >> > > > > > * source code tag "v2.2.0-RC4" [5], > > > > > >>> >> > > > > > * website pull request listing the release and > > > > > publishing > > > > > >>> the > > > > > >>> >> API > > > > > >>> >> > > > > > reference manual [6]. > > > > > >>> >> > > > > > * Java artifacts were built with Maven 3.5.0 and > > > > > >>> OpenJDK/Oracle > > > > > >>> >> > JDK > > > > > >>> >> > > > > > 1.8.0_144. > > > > > >>> >> > > > > > * Python artifacts are deployed along with the > > > source > > > > > >>> release > > > > > >>> >> to > > > > > >>> >> > > the > > > > > >>> >> > > > > > dist.apache.org [2]. > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > The vote will be open for at least 72 hours. It is > > > > > adopted by > > > > > >>> >> > > majority > > > > > >>> >> > > > > > approval, with at least 3 PMC affirmative votes. > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > Thanks, > > > > > >>> >> > > > > > Reuven > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > [1] https://issues.apache.org/ > > > > > jira/secure/ReleaseNote.jspa?p > > > > > >>> >> > > > > > rojectId=12319527&version=12341044 > > > > > >>> >> > > > > > [2] https://dist.apache.org/repos/ > > > dist/dev/beam/2.2.0/ > > > > > >>> >> > > > > > [3] > > > > https://dist.apache.org/repos/dist/release/beam/KEYS > > > > > >>> >> > > > > > [4] https://repository.apache.org/ > > > content/repositories/ > > > > > >>> >> > > > > orgapachebeam-1025/ > > > > > >>> >> > > > > > [5] > https://github.com/apache/beam/tree/v2.2.0-RC4 > > > > > >>> >> > > > > > <https://github.com/apache/beam/tree/v2.2.0-RC4> > > > > > >>> >> > > > > > [6] https://github.com/apache/beam-site/pull/337 > > > > > >>> >> > > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > >>> >> > > > > > > > >>> >> > > > > > > >>> >> > > > > > >>> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > -- Thanks & Regards, Nishu Tayal
