Thanks Nishu. So, if I understand correctly, your pipelines were running on non-YARN, but you're planning to run with YARN?
I meanwhile was able to get Flink running on Dataproc (YARN), and validated quickstart and game examples. At this point we need validation for Spark and Flink non-YARN [I think if Nishu's runs were non-YARN, they'd give us enough confidence, combined with the success of other validations of Spark and Flink runners?], and Apex on YARN. However, it seems that in previous RCs we were not validating Apex on YARN, only local cluster. Is it needed this time? On Wed, Nov 22, 2017 at 12:28 PM Nishu <[email protected]> wrote: > Hi Eugene, > > No, I didn't try with those instead I have my custom pipeline where Kafka > topic is the source. I have defined a Global Window and processing time > trigger to read the data. Further it runs some transformation i.e. > GroupByKey and CoGroupByKey. on the windowed collections. > I was running the same pipeline on direct runner and spark runner earlier.. > Today gave it a try with Flink on Yarn. > > Best Regards, > Nishu. > > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon > > > Virus-free. > www.avast.com > < > https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > On Wed, Nov 22, 2017 at 8:07 PM, Eugene Kirpichov < > [email protected]> wrote: > > > Thanks Nishu! Can you clarify which pipeline you were running? > > The validation spreadsheet includes 1) the quickstart and 2) mobile game > > walkthroughs. Was it one of these, or your custom pipeline? > > > > On Wed, Nov 22, 2017 at 10:20 AM Nishu <[email protected]> wrote: > > > > > Hi, > > > > > > Typo in previous mail. I meant Flink runner. > > > > > > Thanks, > > > Nishu > > > On Wed, 22 Nov 2017 at 19.17, > > > > > > > Hi, > > > > > > > > I build a pipeline using RC 2.2 today and ran with runner on yarn. > > > > It worked seamlessly for unbounded sources. Couldn’t see any issues > > with > > > > my pipeline so far :) > > > > > > > > > > > > Thanks,Nishu > > > > > > > > On Wed, 22 Nov 2017 at 18.57, Reuven Lax <[email protected]> > > > wrote: > > > > > > > >> Who is validating Flink and Yarn? > > > >> > > > >> On Tue, Nov 21, 2017 at 9:26 AM, Kenneth Knowles > > <[email protected] > > > > > > > >> wrote: > > > >> > > > >> > On Mon, Nov 20, 2017 at 5:01 PM, Eugene Kirpichov < > > > >> > [email protected]> wrote: > > > >> > > > > >> > > In the verification spreadsheet, I'm not sure I understand the > > > >> difference > > > >> > > between the "YARN" and "Standalone cluster/service". Which is > > > >> Dataproc? > > > >> > It > > > >> > > definitely uses YARN, but it is also a standalone > cluster/service. > > > >> Does > > > >> > it > > > >> > > count for both? > > > >> > > > > > >> > > > > >> > No, it doesn't. A number of runners have their own non-YARN > cluster > > > >> mode. I > > > >> > would expect that the launching experience might be different and > > the > > > >> > portable container management to differ. If they are identical, > > > experts > > > >> in > > > >> > those systems should feel free to coalesce the rows. Conversely, > as > > > >> other > > > >> > platforms become supported, they could be added or not based on > > > whether > > > >> > they are substantively different from a user experience or QA > point > > of > > > >> > view. > > > >> > > > > >> > Kenn > > > >> > > > > >> > > > > >> > > Seems now we're missing just Apex and Flink cluster > verifications. > > > >> > > > > > >> > > *though Spark runner took 6x longer to run UserScore, partially > I > > > >> guess > > > >> > > because it didn't do autoscaling (Dataflow runner ramped up to 5 > > > >> workers > > > >> > > whereas Spark runner used 2 workers). For some reason Spark > runner > > > >> chose > > > >> > > not to split the 10GB input files into chunks. > > > >> > > > > > >> > > On Mon, Nov 20, 2017 at 3:46 PM Reuven Lax > > <[email protected] > > > > > > > >> > > wrote: > > > >> > > > > > >> > > > Done > > > >> > > > > > > >> > > > On Tue, Nov 21, 2017 at 3:08 AM, Robert Bradshaw < > > > >> > > > [email protected]> wrote: > > > >> > > > > > > >> > > > > Thanks. You need to re-sign as well. > > > >> > > > > > > > >> > > > > On Mon, Nov 20, 2017 at 12:14 AM, Reuven Lax > > > >> > <[email protected] > > > >> > > > > > > >> > > > > wrote: > > > >> > > > > > FYI these generated files have been removed from the > source > > > >> > > > distribution. > > > >> > > > > > > > > >> > > > > > On Sat, Nov 18, 2017 at 9:09 AM, Reuven Lax < > > [email protected] > > > > > > > >> > > wrote: > > > >> > > > > > > > > >> > > > > >> hmmm, I thought I removed those generated files from the > > zip > > > >> file > > > >> > > > before > > > >> > > > > >> sending this email. Let me check again. > > > >> > > > > >> > > > >> > > > > >> Reuven > > > >> > > > > >> > > > >> > > > > >> On Sat, Nov 18, 2017 at 8:52 AM, Robert Bradshaw < > > > >> > > > > >> [email protected]> wrote: > > > >> > > > > >> > > > >> > > > > >>> The source distribution contains a couple of files not > on > > > >> github > > > >> > > > (e.g. > > > >> > > > > >>> folders that were added on master, Python generated > > files). > > > >> The > > > >> > pom > > > >> > > > > >>> files differed only by missing -SNAPSHOT, other than > that > > > >> > > presumably > > > >> > > > > >>> the source release should just be "wget > > > >> > > > > >>> > https://github.com/apache/beam/archive/release-2.2.0.zip > > "? > > > >> > > > > >>> > > > >> > > > > >>> diff -rq apache-beam-2.2.0 beam/ | grep -v pom.xml > > > >> > > > > >>> > > > >> > > > > >>> # OK? > > > >> > > > > >>> > > > >> > > > > >>> Only in apache-beam-2.2.0: DEPENDENCIES > > > >> > > > > >>> > > > >> > > > > >>> # Expected. > > > >> > > > > >>> > > > >> > > > > >>> Only in beam/: .git > > > >> > > > > >>> Only in beam/: .gitattributes > > > >> > > > > >>> Only in beam/: .gitignore > > > >> > > > > >>> > > > >> > > > > >>> # These folders are probably from switching around > between > > > >> master > > > >> > > and > > > >> > > > > >>> git branches. > > > >> > > > > >>> > > > >> > > > > >>> Only in apache-beam-2.2.0: model > > > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: examples > > > >> > > > > >>> Only in apache-beam-2.2.0/runners/flink: runner > > > >> > > > > >>> Only in apache-beam-2.2.0/runners/gearpump: jarstore > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: gcp-core > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/extensions: > sketching > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: > > file-based-io-tests > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/io: hdfs > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > > > >> > maven-archetypes/examples/src/ > > > >> > > ma > > > >> > > > > >>> in/resources/archetype-resources: > > > >> > > > > >>> src > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java/ > > > >> maven-archetypes/examples- > > > >> > > java8/ > > > >> > > > > >>> src/main/resources/archetype-resources: > > > >> > > > > >>> src > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/java: microbenchmarks > > > >> > > > > >>> > > > >> > > > > >>> # Here's the generated protos. > > > >> > > > > >>> > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_artifact_api_pb2_grpc.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_artifact_api_pb2.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_fn_api_pb2_grpc.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_fn_api_pb2.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_job_api_pb2_grpc.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_job_api_pb2.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_provision_api_pb2_grpc.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_provision_api_pb2.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_runner_api_pb2_grpc.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> beam_runner_api_pb2.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> endpoints_pb2_grpc.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> endpoints_pb2.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> standard_window_fns_pb2_grpc.py > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python/ > > > >> > apache_beam/portability/api: > > > >> > > > > >>> standard_window_fns_pb2.py > > > >> > > > > >>> > > > >> > > > > >>> And some other sdist generated Python files. > > > >> > > > > >>> > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: .eggs > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: LICENSE > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: NOTICE > > > >> > > > > >>> Only in apache-beam-2.2.0/sdks/python: README.md > > > >> > > > > >>> > > > >> > > > > >>> Presumably we should just purge these files from the rc? > > > >> > > > > >>> > > > >> > > > > >>> > > > >> > > > > >>> FWIW, the Python tarball looks fine. > > > >> > > > > >>> > > > >> > > > > >>> On Fri, Nov 17, 2017 at 4:40 PM, Eugene Kirpichov > > > >> > > > > >>> <[email protected]> wrote: > > > >> > > > > >>> > How can I specify a dependency on the staged RC? E.g. > > I'm > > > >> > trying > > > >> > > to > > > >> > > > > >>> > validate the quickstart per > > > >> > > > > >>> > https://beam.apache.org/get-started/quickstart-java/ > > and > > > >> > > > specifying > > > >> > > > > >>> version > > > >> > > > > >>> > 2.2.0 doesn't work I suppose because it's not released > > > yet. > > > >> > > Should > > > >> > > > I > > > >> > > > > >>> pass > > > >> > > > > >>> > some command-line flag to mvn to make it fetch the > > version > > > >> from > > > >> > > the > > > >> > > > > >>> staging > > > >> > > > > >>> > area? > > > >> > > > > >>> > > > > >> > > > > >>> > On Fri, Nov 17, 2017 at 4:37 PM Lukasz Cwik > > > >> > > > <[email protected] > > > >> > > > > > > > > >> > > > > >>> > wrote: > > > >> > > > > >>> > > > > >> > > > > >>> >> Its open to all, its just that there are binding > votes > > > and > > > >> > > > > non-binding > > > >> > > > > >>> >> votes. > > > >> > > > > >>> >> > > > >> > > > > >>> >> On Fri, Nov 17, 2017 at 4:26 PM, Valentyn Tymofieiev > < > > > >> > > > > >>> >> [email protected]> wrote: > > > >> > > > > >>> >> > > > >> > > > > >>> >> > I have a process question: is the vote open for > > > >> committers > > > >> > > only > > > >> > > > or > > > >> > > > > >>> for > > > >> > > > > >>> >> all > > > >> > > > > >>> >> > contributors? > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > On Fri, Nov 17, 2017 at 4:06 PM, Lukasz Cwik > > > >> > > > > >>> <[email protected]> > > > >> > > > > >>> >> > wrote: > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > +1, Approve the release > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > I have verified the wordcount quickstart on the > > > Apache > > > >> > Beam > > > >> > > > > website > > > >> > > > > >>> >> using > > > >> > > > > >>> >> > > Apex, DirectRunner, Flink & Spark on Linux. > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > The Gearpump runner is yet to have a quickstart > > > listed > > > >> on > > > >> > > our > > > >> > > > > >>> website. > > > >> > > > > >>> >> > > Adding the quickstart is already represented by > > this > > > >> > > existing > > > >> > > > > >>> issue: > > > >> > > > > >>> >> > > https://issues.apache.org/jira/browse/BEAM-2692 > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > On Fri, Nov 17, 2017 at 11:50 AM, Valentyn > > > Tymofieiev < > > > >> > > > > >>> >> > > [email protected]> wrote: > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > I have verified: SHA & MD5 signatures of Python > > > >> > artifacts > > > >> > > in > > > >> > > > > >>> [2], and > > > >> > > > > >>> >> > > > checked Python side of the validation checklist > > on > > > >> > Linux. > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > There is one known issue in UserScore example > for > > > >> > Dataflow > > > >> > > > > >>> runner. > > > >> > > > > >>> >> The > > > >> > > > > >>> >> > > > issue has been fixed on master branch and does > > not > > > >> > > require a > > > >> > > > > >>> >> > cherry-pick > > > >> > > > > >>> >> > > at > > > >> > > > > >>> >> > > > this point. A workaround is to pass > > > >> --save_main_session > > > >> > > > > pipeline > > > >> > > > > >>> >> > option. > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > RC4 looks good to me so far. > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > On Fri, Nov 17, 2017 at 11:30 AM, Kenneth > Knowles > > > >> > > > > >>> >> > <[email protected] > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > wrote: > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > Hi all, > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > Following up on past discussions and > > > >> > > > > >>> >> > > > > https://issues.apache.org/ > > jira/browse/BEAM-1189 > > > I > > > >> > have > > > >> > > > > >>> prepared a > > > >> > > > > >>> >> > > > > spreadsheet so we can sign up for validation > > > steps > > > >> > that > > > >> > > > > must be > > > >> > > > > >>> >> done > > > >> > > > > >>> >> > > by a > > > >> > > > > >>> >> > > > > human. > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > The spreadsheet for 2.2.0 is at > > > >> > > > > >>> >> > > > > > > > https://s.apache.org/beam-2.2.0-release-validation > > > >> . > > > >> > > > > Everyone > > > >> > > > > >>> can > > > >> > > > > >>> >> > edit, > > > >> > > > > >>> >> > > > so > > > >> > > > > >>> >> > > > > it is easy to sign up or add new rows. > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > FYI the template is at > > > >> > > > > >>> >> https://s.apache.org/beam-release-validation. > > > >> > > > > >>> >> > > > After > > > >> > > > > >>> >> > > > > this release, I will update the rows in the > > > >> template > > > >> > to > > > >> > > > > match > > > >> > > > > >>> >> > whatever > > > >> > > > > >>> >> > > we > > > >> > > > > >>> >> > > > > have added to the validation for 2.2.0. > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > Kenn > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > On Thu, Nov 16, 2017 at 10:08 PM, Reuven Lax > > > >> > > > > >>> >> > <[email protected] > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > wrote: > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > Hi everyone, > > > >> > > > > >>> >> > > > > > > > > >> > > > > >>> >> > > > > > Please review and vote on the release > > candidate > > > >> #4 > > > >> > for > > > >> > > > the > > > >> > > > > >>> >> version > > > >> > > > > >>> >> > > > 2.2.0, > > > >> > > > > >>> >> > > > > > as follows: > > > >> > > > > >>> >> > > > > > [ ] +1, Approve the release > > > >> > > > > >>> >> > > > > > [ ] -1, Do not approve the release > (please > > > >> provide > > > >> > > > > specific > > > >> > > > > >>> >> > > comments) > > > >> > > > > >>> >> > > > > > > > > >> > > > > >>> >> > > > > > > > > >> > > > > >>> >> > > > > > The complete staging area is available for > > your > > > >> > > review, > > > >> > > > > which > > > >> > > > > >>> >> > > includes: > > > >> > > > > >>> >> > > > > > * JIRA release notes [1], > > > >> > > > > >>> >> > > > > > * the official Apache source release to > be > > > >> > deployed > > > >> > > to > > > >> > > > > >>> >> > > > dist.apache.org > > > >> > > > > >>> >> > > > > > [2], > > > >> > > > > >>> >> > > > > > which is signed with the key with > fingerprint > > > >> > B98B7708 > > > >> > > > > [3], > > > >> > > > > >>> >> > > > > > * all artifacts to be deployed to the > Maven > > > >> > Central > > > >> > > > > >>> Repository > > > >> > > > > >>> >> > [4], > > > >> > > > > >>> >> > > > > > * source code tag "v2.2.0-RC4" [5], > > > >> > > > > >>> >> > > > > > * website pull request listing the > release > > > and > > > >> > > > > publishing > > > >> > > > > >>> the > > > >> > > > > >>> >> API > > > >> > > > > >>> >> > > > > > reference manual [6]. > > > >> > > > > >>> >> > > > > > * Java artifacts were built with Maven > > 3.5.0 > > > >> and > > > >> > > > > >>> OpenJDK/Oracle > > > >> > > > > >>> >> > JDK > > > >> > > > > >>> >> > > > > > 1.8.0_144. > > > >> > > > > >>> >> > > > > > * Python artifacts are deployed along > with > > > the > > > >> > > source > > > >> > > > > >>> release > > > >> > > > > >>> >> to > > > >> > > > > >>> >> > > the > > > >> > > > > >>> >> > > > > > dist.apache.org [2]. > > > >> > > > > >>> >> > > > > > > > > >> > > > > >>> >> > > > > > The vote will be open for at least 72 > hours. > > It > > > >> is > > > >> > > > > adopted by > > > >> > > > > >>> >> > > majority > > > >> > > > > >>> >> > > > > > approval, with at least 3 PMC affirmative > > > votes. > > > >> > > > > >>> >> > > > > > > > > >> > > > > >>> >> > > > > > Thanks, > > > >> > > > > >>> >> > > > > > Reuven > > > >> > > > > >>> >> > > > > > > > > >> > > > > >>> >> > > > > > [1] https://issues.apache.org/ > > > >> > > > > jira/secure/ReleaseNote.jspa?p > > > >> > > > > >>> >> > > > > > rojectId=12319527&version=12341044 > > > >> > > > > >>> >> > > > > > [2] https://dist.apache.org/repos/ > > > >> > > dist/dev/beam/2.2.0/ > > > >> > > > > >>> >> > > > > > [3] > > > >> > > > https://dist.apache.org/repos/dist/release/beam/KEYS > > > >> > > > > >>> >> > > > > > [4] https://repository.apache.org/ > > > >> > > content/repositories/ > > > >> > > > > >>> >> > > > > orgapachebeam-1025/ > > > >> > > > > >>> >> > > > > > [5] https://github.com/apache/ > > > >> beam/tree/v2.2.0-RC4 > > > >> > > > > >>> >> > > > > > < > > > https://github.com/apache/beam/tree/v2.2.0-RC4> > > > >> > > > > >>> >> > > > > > [6] > > > https://github.com/apache/beam-site/pull/337 > > > >> > > > > >>> >> > > > > > > > > >> > > > > >>> >> > > > > > > > >> > > > > >>> >> > > > > > > >> > > > > >>> >> > > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > >> > > > > >>> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > -- > > > > Thanks & Regards, > > > > Nishu Tayal > > > > > > > > > > < > > > https://www.avast.com/sig-email?utm_medium=email&utm_ > > source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon > > > > > > > Virus-free. > > > www.avast.com > > > < > > > https://www.avast.com/sig-email?utm_medium=email&utm_ > > source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link > > > > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > > > > > > > > > -- > Thanks & Regards, > Nishu Tayal >
