BTW, did you notice that doing a shade now logs something like: [WARNING] beam-model-fn-execution-2.7.0.jar, beam-model-job-management-2.7.0.jar, beam-model-pipeline-2.7.0.jar define 6660 overlapping classes: [WARNING] - org.apache.beam.vendor.netty.v4.io.netty.handler.codec.http.HttpClientCodec$1 [WARNING] - org.apache.beam.vendor.guava.v20.com.google.common.util.concurrent.AggregateFutureState$SafeAtomicHelper [WARNING] - org.apache.beam.vendor.netty.v4.io.netty.util.concurrent.DefaultFutureListeners [WARNING] - org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.OpenSslSessionContext$1 [WARNING] - org.apache.beam.vendor.netty.v4.io.netty.handler.ssl.Java9SslUtils$4 [WARNING] - org.apache.beam.vendor.guava.v20.com.google.common.collect.ImmutableMultimap$Builder [WARNING] - org.apache.beam.vendor.netty.v4.io.netty.handler.codec.spdy.SpdyHeaders [WARNING] - org.apache.beam.vendor.protobuf.v3.com.google.protobuf.DescriptorProtos$FieldDescriptorProtoOrBuilder [WARNING] - org.apache.beam.vendor.guava.v20.com.google.common.collect.AbstractMultimap [WARNING] - org.apache.beam.vendor.guava.v20.com.google.common.io.BaseEncoding$3 [WARNING] - 6650 more...
Looks like the new shading policy impl was merged a bit too fast ;) Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> Le mar. 11 sept. 2018 à 21:42, Jean-Baptiste Onofré <j...@nanthrax.net> a écrit : > I'm taking the Spark runner one. > > Regards > JB > > On 11/09/2018 21:15, Ahmet Altay wrote: > > Could anyone else help with looking at these issues earlier? > > > > On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau > > <rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote: > > > > Im running this main [1] through this IT [2]. Was working fine since > > ~1 year but 2.7.0 broke it. Didnt investigate more but can have a > > look later this month if it helps. > > > > [1] > > > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java > > < > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java > > > > [2] > > > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java > > < > https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java > > > > > > Le mar. 11 sept. 2018 20:54, Charles Chen <c...@google.com > > <mailto:c...@google.com>> a écrit : > > > > Romain: can you give more details on the failure you're > > encountering, i.e. how you are performing this validation? > > > > On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste Onofré > > <j...@nanthrax.net <mailto:j...@nanthrax.net>> wrote: > > > > Hi, > > > > weird, I didn't have it on Beam samples. Let me try to > > reproduce and I > > will create the Jira. > > > > Regards > > JB > > > > On 11/09/2018 11:44, Romain Manni-Bucau wrote: > > > -1, seems spark integration is broken (tested with spark > > 2.3.1 and 2.2.1): > > > > > > 18/09/11 11:33:29 WARN TaskSetManager: Lost task 0.0 in > > stage 0.0 (TID 0, RMANNIBUCAU, executor 0): > > java.lang.ClassCastException: cannot assign instance of > > scala.collection.immutable.List$SerializationProxy to > > fieldorg.apache.spark.rdd.RDD.org > > <http://fieldorg.apache.spark.rdd.RDD.org> > > <http://org.apache.spark.rdd.RDD.org > > <http://org.apache.spark.rdd.RDD.org > >>$apache$spark$rdd$RDD$$dependencies_ > > of type scala.collection.Seq in instance of > > org.apache.spark.rdd.MapPartitionsRDD > > > at > > > > java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233) > > > > > > > > > Also the issue Lukasz identified is important even if > > workarounds can be > > > put in place so +1 to fix it as well if possible. > > > > > > Romain Manni-Bucau > > > @rmannibucau <https://twitter.com/rmannibucau > > <https://twitter.com/rmannibucau>> | Blog > > > <https://rmannibucau.metawerx.net/ > > <https://rmannibucau.metawerx.net/>> | Old Blog > > > <http://rmannibucau.wordpress.com > > <http://rmannibucau.wordpress.com>> | Github > > > <https://github.com/rmannibucau > > <https://github.com/rmannibucau>> | LinkedIn > > > <https://www.linkedin.com/in/rmannibucau > > <https://www.linkedin.com/in/rmannibucau>> | Book > > > > > < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > < > https://www.packtpub.com/application-development/java-ee-8-high-performance > >> > > > > > > > > > Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik > > <lc...@google.com <mailto:lc...@google.com> > > > <mailto:lc...@google.com <mailto:lc...@google.com>>> a > > écrit : > > > > > > I found an issue where we are no longer packaging the > > pom.xml within > > > the artifact jars at > > META-INF/maven/groupId/artifactId. More details > > > in https://issues.apache.org/jira/browse/BEAM-5351 > > <https://issues.apache.org/jira/browse/BEAM-5351>. I > wouldn't > > > consider this a blocker but it was an easy fix > > > (https://github.com/apache/beam/pull/6358 > > <https://github.com/apache/beam/pull/6358>) and users may > > rely on the > > > pom.xml. > > > > > > Should we recut the release candidate to include this? > > > > > > On Mon, Sep 10, 2018 at 4:58 AM Jean-Baptiste Onofré > > > <j...@nanthrax.net <mailto:j...@nanthrax.net> > > <mailto:j...@nanthrax.net <mailto:j...@nanthrax.net>>> wrote: > > > > > > +1 (binding) > > > > > > Tested successfully on Beam Samples. > > > > > > Thanks ! > > > > > > Regards > > > JB > > > > > > On 07/09/2018 23:56, Charles Chen wrote: > > > > Hi everyone, > > > > > > > > Please review and vote on the release > > candidate #1 for the > > > version > > > > 2.7.0, as follows: > > > > [ ] +1, Approve the release > > > > [ ] -1, Do not approve the release (please > > provide specific > > > comments) > > > > > > > > The complete staging area is available for > > your review, which > > > includes: > > > > * JIRA release notes [1], > > > > * the official Apache source release to be > > deployed to > > > dist.apache.org <http://dist.apache.org> > > <http://dist.apache.org> > > > > <http://dist.apache.org> [2], which is signed > > with the key with > > > > fingerprint 45C60AAAD115F560 [3], > > > > * all artifacts to be deployed to the Maven > > Central > > > Repository [4], > > > > * source code tag "v2.7.0-RC1" [5], > > > > * website pull request listing the release and > > publishing the API > > > > reference manual [6]. > > > > * Java artifacts were built with Gradle 4.8 > > and OpenJDK > > > > 1.8.0_181-8u181-b13-1~deb9u1-b13. > > > > * Python artifacts are deployed along with the > > source release > > > to the > > > > dist.apache.org <http://dist.apache.org> > > <http://dist.apache.org> > > > <http://dist.apache.org> [2]. > > > > > > > > The vote will be open for at least 72 hours. > > It is adopted by > > > majority > > > > approval, with at least 3 PMC affirmative > votes. > > > > > > > > Thanks, > > > > Charles > > > > > > > > [1] > > > > > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654 > > < > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654 > > > > > > [2] > > https://dist.apache.org/repos/dist/dev/beam/2.7.0 > > <https://dist.apache.org/repos/dist/dev/beam/2.7.0> > > > > [3] > > https://dist.apache.org/repos/dist/dev/beam/KEYS > > <https://dist.apache.org/repos/dist/dev/beam/KEYS> > > > > [4] > > > > > > https://repository.apache.org/content/repositories/orgapachebeam-1046/ > > < > https://repository.apache.org/content/repositories/orgapachebeam-1046/> > > > > [5] > > https://github.com/apache/beam/tree/v2.7.0-RC1 > > <https://github.com/apache/beam/tree/v2.7.0-RC1> > > > > [6] > > https://github.com/apache/beam-site/pull/549 > > <https://github.com/apache/beam-site/pull/549> > > > > > > -- > > > Jean-Baptiste Onofré > > > jbono...@apache.org <mailto:jbono...@apache.org> > > <mailto:jbono...@apache.org <mailto:jbono...@apache.org>> > > > http://blog.nanthrax.net > > > Talend - http://www.talend.com > > > > > > > >
