Le mar. 18 sept. 2018 à 16:44, Jean-Baptiste Onofré <j...@nanthrax.net> a écrit :
> Hi, > > I don't have the issue ;) > > As said in my vote, I tested 2.7.0 RC1 on beam-samples with Spark > without problem. > > I don't reproduce Romain issue as well. > > @Romain can you provide some details to reproduce the issue ? > Sure, you can use this reproducer: https://github.com/rmannibucau/beam-2.7.0-fails It shows that it suceeds on 2.6 and fails on 2.7. > > Regards > JB > > On 17/09/2018 19:17, Charles Chen wrote: > > Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the > > release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)? > > > > Romain, JB: is there any way I can help with debugging the issue you're > > facing so we can unblock the release? > > > > On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rang...@google.com > > <mailto:rang...@google.com>> wrote: > > > > I would like propose one more cherrypick for RC2 > > : https://github.com/apache/beam/pull/6391 > > This is a KafkaIO bug fix. Once a user hits this bug, there is no > > easy work around for them, especially on Dataflow. Only work around > > in Dataflow is to restart or reload the job. > > > > The fix itself fairly safe and is tested. > > Raghu. > > > > On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko > > <aromanenko....@gmail.com <mailto:aromanenko....@gmail.com>> wrote: > > > > Perhaps it could help, but I run simple WordCount (built with > > Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine > > for me. > > > >> On 14 Sep 2018, at 06:56, Romain Manni-Bucau > >> <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: > >> > >> Hi Charles, > >> > >> I didn't get enough time to check deeply but it is clearly a > >> dependency issue and it is not in beam spark runner itself but > >> in another transitive module of beam. It does not happen in > >> existing spark test cause none of them are in a cluster (even > >> just with 1 worker) but this seems to be a regression since > >> 2.6 works OOTB. > >> > >> Romain Manni-Bucau > >> @rmannibucau <https://twitter.com/rmannibucau> | Blog > >> <https://rmannibucau.metawerx.net/> | Old Blog > >> <http://rmannibucau.wordpress.com/> | Github > >> <https://github.com/rmannibucau> | LinkedIn > >> <https://www.linkedin.com/in/rmannibucau> | Book > >> < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > >> > >> > >> Le jeu. 13 sept. 2018 à 22:15, Charles Chen <c...@google.com > >> <mailto:c...@google.com>> a écrit : > >> > >> Romain and JB, can you please add the results of your > >> investigations into the errors you've seen above? Given > >> that the existing SparkRunner tests pass for this RC, and > >> that the integration test you ran is in another repo that > >> is not continuously tested with Beam, it is not clear how > >> we should move forward and whether this is a blocking > >> issue, unless we can find a root cause in Beam. > >> > >> On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot > >> <echauc...@apache.org <mailto:echauc...@apache.org>> wrote: > >> > >> Hi all, > >> > >> on a performance and functional regression stand point > >> I see no regression: > >> > >> I looked at nexmark graphs "output pcollection size" > >> and "execution time" around release cut date on > >> dataflow, spark, flink and direct runner in batch and > >> streaming modes. There seems to be no regression. > >> > >> Etienne > >> > >> Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen > >> a écrit : > >>> The SparkRunner validation test > >>> (here: > https://beam.apache.org/contribute/release-guide/#run-validation-tests) > >>> passes on my machine. It looks like we are likely > >>> missing test coverage where Romain is hitting issues. > >>> > >>> On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay > >>> <al...@google.com <mailto:al...@google.com>> wrote: > >>>> Could anyone else help with looking at these issues > >>>> earlier? > >>>> > >>>> On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau > >>>> <rmannibu...@gmail.com > >>>> <mailto:rmannibu...@gmail.com>> wrote: > >>>>> Im running this main [1] through this IT [2]. Was > >>>>> working fine since ~1 year but 2.7.0 broke it. > >>>>> Didnt investigate more but can have a look later > >>>>> this month if it helps. > >>>>> > >>>>> [1] > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java > >>>>> [2] > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java > >>>>> > >>>>> Le mar. 11 sept. 2018 20:54, Charles Chen > >>>>> <c...@google.com <mailto:c...@google.com>> a écrit : > >>>>>> Romain: can you give more details on the failure > >>>>>> you're encountering, i.e. how you are performing > >>>>>> this validation? > >>>>>> > >>>>>> On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste > >>>>>> Onofré <j...@nanthrax.net <mailto:j...@nanthrax.net>> > >>>>>> wrote: > >>>>>>> Hi, > >>>>>>> > >>>>>>> weird, I didn't have it on Beam samples. Let me > >>>>>>> try to reproduce and I > >>>>>>> will create the Jira. > >>>>>>> > >>>>>>> Regards > >>>>>>> JB > >>>>>>> > >>>>>>> On 11/09/2018 11:44, Romain Manni-Bucau wrote: > >>>>>>> > -1, seems spark integration is broken (tested > >>>>>>> with spark 2.3.1 and 2.2.1): > >>>>>>> > > >>>>>>> > 18/09/11 11:33:29 WARN TaskSetManager: Lost > >>>>>>> task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU, > >>>>>>> executor 0): java.lang.ClassCastException: cannot > >>>>>>> assign instance of > >>>>>>> scala.collection.immutable.List$SerializationProxy > to > >>>>>>> fieldorg.apache.spark.rdd.RDD.org > >>>>>>> <http://fieldorg.apache.spark.rdd.rdd.org/> > >>>>>>> <http://org.apache.spark.rdd.RDD.org > >>>>>>> <http://org.apache.spark.rdd.rdd.org/ > >>$apache$spark$rdd$RDD$$dependencies_ > >>>>>>> of type scala.collection.Seq in instance of > >>>>>>> org.apache.spark.rdd.MapPartitionsRDD > >>>>>>> > at > >>>>>>> > > java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233) > >>>>>>> > > >>>>>>> > > >>>>>>> > Also the issue Lukasz identified is important > >>>>>>> even if workarounds can be > >>>>>>> > put in place so +1 to fix it as well if possible. > >>>>>>> > > >>>>>>> > Romain Manni-Bucau > >>>>>>> > @rmannibucau <https://twitter.com/rmannibucau> > >>>>>>> | Blog > >>>>>>> > <https://rmannibucau.metawerx.net/> | Old Blog > >>>>>>> > <http://rmannibucau.wordpress.com > >>>>>>> <http://rmannibucau.wordpress.com/>> | Github > >>>>>>> > <https://github.com/rmannibucau> | LinkedIn > >>>>>>> > <https://www.linkedin.com/in/rmannibucau> | Book > >>>>>>> > > >>>>>>> < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > >>>>>>> > > >>>>>>> > > >>>>>>> > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik > >>>>>>> <lc...@google.com <mailto:lc...@google.com> > >>>>>>> > <mailto:lc...@google.com > >>>>>>> <mailto:lc...@google.com>>> a écrit : > >>>>>>> > > >>>>>>> > I found an issue where we are no longer > >>>>>>> packaging the pom.xml within > >>>>>>> > the artifact jars at > >>>>>>> META-INF/maven/groupId/artifactId. More details > >>>>>>> > in > >>>>>>> https://issues.apache.org/jira/browse/BEAM-5351. > >>>>>>> I wouldn't > >>>>>>> > consider this a blocker but it was an easy > fix > >>>>>>> > (https://github.com/apache/beam/pull/6358) > >>>>>>> and users may rely on the > >>>>>>> > pom.xml. > >>>>>>> > > >>>>>>> > Should we recut the release candidate to > >>>>>>> include this? > >>>>>>> > > >>>>>>> > On Mon, Sep 10, 2018 at 4:58 AM > >>>>>>> Jean-Baptiste Onofré > >>>>>>> > <j...@nanthrax.net <mailto:j...@nanthrax.net> > >>>>>>> <mailto:j...@nanthrax.net > >>>>>>> <mailto:j...@nanthrax.net>>> wrote: > >>>>>>> > > >>>>>>> > +1 (binding) > >>>>>>> > > >>>>>>> > Tested successfully on Beam Samples. > >>>>>>> > > >>>>>>> > Thanks ! > >>>>>>> > > >>>>>>> > Regards > >>>>>>> > JB > >>>>>>> > > >>>>>>> > On 07/09/2018 23:56, Charles Chen wrote: > >>>>>>> > > Hi everyone, > >>>>>>> > > > >>>>>>> > > Please review and vote on the > >>>>>>> release candidate #1 for the > >>>>>>> > version > >>>>>>> > > 2.7.0, as follows: > >>>>>>> > > [ ] +1, Approve the release > >>>>>>> > > [ ] -1, Do not approve the release > >>>>>>> (please provide specific > >>>>>>> > comments) > >>>>>>> > > > >>>>>>> > > The complete staging area is > >>>>>>> available for your review, which > >>>>>>> > includes: > >>>>>>> > > * JIRA release notes [1], > >>>>>>> > > * the official Apache source release > >>>>>>> to be deployed to > >>>>>>> > dist.apache.org > >>>>>>> <http://dist.apache.org/> <http://dist.apache.org > >>>>>>> <http://dist.apache.org/>> > >>>>>>> > > <http://dist.apache.org > >>>>>>> <http://dist.apache.org/>> [2], which is signed > >>>>>>> with the key with > >>>>>>> > > fingerprint 45C60AAAD115F560 [3], > >>>>>>> > > * all artifacts to be deployed to > >>>>>>> the Maven Central > >>>>>>> > Repository [4], > >>>>>>> > > * source code tag "v2.7.0-RC1" [5], > >>>>>>> > > * website pull request listing the > >>>>>>> release and publishing the API > >>>>>>> > > reference manual [6]. > >>>>>>> > > * Java artifacts were built with > >>>>>>> Gradle 4.8 and OpenJDK > >>>>>>> > > 1.8.0_181-8u181-b13-1~deb9u1-b13. > >>>>>>> > > * Python artifacts are deployed > >>>>>>> along with the source release > >>>>>>> > to the > >>>>>>> > > dist.apache.org > >>>>>>> <http://dist.apache.org/> <http://dist.apache.org > >>>>>>> <http://dist.apache.org/>> > >>>>>>> > <http://dist.apache.org > >>>>>>> <http://dist.apache.org/>> [2]. > >>>>>>> > > > >>>>>>> > > The vote will be open for at least > >>>>>>> 72 hours. It is adopted by > >>>>>>> > majority > >>>>>>> > > approval, with at least 3 PMC > >>>>>>> affirmative votes. > >>>>>>> > > > >>>>>>> > > Thanks, > >>>>>>> > > Charles > >>>>>>> > > > >>>>>>> > > [1] > >>>>>>> > > > >>>>>>> > > >>>>>>> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654 > >>>>>>> > > [2] > >>>>>>> https://dist.apache.org/repos/dist/dev/beam/2.7.0 > >>>>>>> > > [3] > >>>>>>> https://dist.apache.org/repos/dist/dev/beam/KEYS > >>>>>>> > > [4] > >>>>>>> > > >>>>>>> > https://repository.apache.org/content/repositories/orgapachebeam-1046/ > >>>>>>> > > [5] > >>>>>>> https://github.com/apache/beam/tree/v2.7.0-RC1 > >>>>>>> > > [6] > >>>>>>> https://github.com/apache/beam-site/pull/549 > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Jean-Baptiste Onofré > >>>>>>> > jbono...@apache.org > >>>>>>> <mailto:jbono...@apache.org> > >>>>>>> <mailto:jbono...@apache.org > >>>>>>> <mailto:jbono...@apache.org>> > >>>>>>> > http://blog.nanthrax.net > >>>>>>> <http://blog.nanthrax.net/> > >>>>>>> > Talend - http://www.talend.com > >>>>>>> <http://www.talend.com/> > >>>>>>> > > >>>> > > > > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com >
