Hi Charles,
Thanks for driving the release. Here are the cherrypicks I mentioned:
https://github.com/apache/beam/pull/6427
Thanks,
Max
On 17.09.18 19:17, Charles Chen wrote:
Luke, Maximillian, Raghu, can you please propose cherry-pick PRs to the
release-2.7.0 for your issues and add me as a reviewer (@charlesccychen)?
Romain, JB: is there any way I can help with debugging the issue you're
facing so we can unblock the release?
On Fri, Sep 14, 2018 at 1:49 PM Raghu Angadi <rang...@google.com
<mailto:rang...@google.com>> wrote:
I would like propose one more cherrypick for RC2 :
https://github.com/apache/beam/pull/6391
This is a KafkaIO bug fix. Once a user hits this bug, there is no
easy work around for them, especially on Dataflow. Only work around
in Dataflow is to restart or reload the job.
The fix itself fairly safe and is tested.
Raghu.
On Fri, Sep 14, 2018 at 12:52 AM Alexey Romanenko
<aromanenko....@gmail.com <mailto:aromanenko....@gmail.com>> wrote:
Perhaps it could help, but I run simple WordCount (built with
Beam 2.7) on YARN/Spark (HDP Sandbox) cluster and it worked fine
for me.
On 14 Sep 2018, at 06:56, Romain Manni-Bucau
<rmannibu...@gmail.com <mailto:rmannibu...@gmail.com>> wrote:
Hi Charles,
I didn't get enough time to check deeply but it is clearly a
dependency issue and it is not in beam spark runner itself but
in another transitive module of beam. It does not happen in
existing spark test cause none of them are in a cluster (even
just with 1 worker) but this seems to be a regression since
2.6 works OOTB.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com/> | Github
<https://github.com/rmannibucau> | LinkedIn
<https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le jeu. 13 sept. 2018 à 22:15, Charles Chen <c...@google.com
<mailto:c...@google.com>> a écrit :
Romain and JB, can you please add the results of your
investigations into the errors you've seen above? Given
that the existing SparkRunner tests pass for this RC, and
that the integration test you ran is in another repo that
is not continuously tested with Beam, it is not clear how
we should move forward and whether this is a blocking
issue, unless we can find a root cause in Beam.
On Wed, Sep 12, 2018 at 2:08 AM Etienne Chauchot
<echauc...@apache.org <mailto:echauc...@apache.org>> wrote:
Hi all,
on a performance and functional regression stand point
I see no regression:
I looked at nexmark graphs "output pcollection size"
and "execution time" around release cut date on
dataflow, spark, flink and direct runner in batch and
streaming modes. There seems to be no regression.
Etienne
Le mardi 11 septembre 2018 à 12:25 -0700, Charles Chen
a écrit :
The SparkRunner validation test (here:
https://beam.apache.org/contribute/release-guide/#run-validation-tests)
passes on my machine. It looks like we are likely
missing test coverage where Romain is hitting issues.
On Tue, Sep 11, 2018 at 12:15 PM Ahmet Altay
<al...@google.com <mailto:al...@google.com>> wrote:
Could anyone else help with looking at these issues
earlier?
On Tue, Sep 11, 2018 at 12:03 PM, Romain Manni-Bucau
<rmannibu...@gmail.com
<mailto:rmannibu...@gmail.com>> wrote:
Im running this main [1] through this IT [2]. Was
working fine since ~1 year but 2.7.0 broke it.
Didnt investigate more but can have a look later
this month if it helps.
[1]
https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/main/java/org/talend/sdk/component/beam/it/clusterserialization/Main.java
[2]
https://github.com/Talend/component-runtime/blob/master/component-runtime-beam/src/it/serialization-over-cluster/src/test/java/org/talend/sdk/component/beam/it/SerializationOverClusterIT.java
Le mar. 11 sept. 2018 20:54, Charles Chen
<c...@google.com <mailto:c...@google.com>> a écrit :
Romain: can you give more details on the failure
you're encountering, i.e. how you are performing
this validation?
On Tue, Sep 11, 2018 at 9:36 AM Jean-Baptiste
Onofré <j...@nanthrax.net <mailto:j...@nanthrax.net>>
wrote:
Hi,
weird, I didn't have it on Beam samples. Let me
try to reproduce and I
will create the Jira.
Regards
JB
On 11/09/2018 11:44, Romain Manni-Bucau wrote:
> -1, seems spark integration is broken (tested
with spark 2.3.1 and 2.2.1):
>
> 18/09/11 11:33:29 WARN TaskSetManager: Lost
task 0.0 in stage 0.0 (TID 0, RMANNIBUCAU,
executor 0): java.lang.ClassCastException: cannot
assign instance of
scala.collection.immutable.List$SerializationProxy to
fieldorg.apache.spark.rdd.RDD.org
<http://fieldorg.apache.spark.rdd.rdd.org/>
<http://org.apache.spark.rdd.RDD.org
<http://org.apache.spark.rdd.rdd.org/>>$apache$spark$rdd$RDD$$dependencies_
of type scala.collection.Seq in instance of
org.apache.spark.rdd.MapPartitionsRDD
> at
java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233)
>
>
> Also the issue Lukasz identified is important
even if workarounds can be
> put in place so +1 to fix it as well if possible.
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau>
| Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com
<http://rmannibucau.wordpress.com/>> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
>
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le lun. 10 sept. 2018 à 20:48, Lukasz Cwik
<lc...@google.com <mailto:lc...@google.com>
> <mailto:lc...@google.com
<mailto:lc...@google.com>>> a écrit :
>
> I found an issue where we are no longer
packaging the pom.xml within
> the artifact jars at
META-INF/maven/groupId/artifactId. More details
> in
https://issues.apache.org/jira/browse/BEAM-5351.
I wouldn't
> consider this a blocker but it was an easy fix
> (https://github.com/apache/beam/pull/6358)
and users may rely on the
> pom.xml.
>
> Should we recut the release candidate to
include this?
>
> On Mon, Sep 10, 2018 at 4:58 AM
Jean-Baptiste Onofré
> <j...@nanthrax.net <mailto:j...@nanthrax.net>
<mailto:j...@nanthrax.net
<mailto:j...@nanthrax.net>>> wrote:
>
> +1 (binding)
>
> Tested successfully on Beam Samples.
>
> Thanks !
>
> Regards
> JB
>
> On 07/09/2018 23:56, Charles Chen wrote:
> > Hi everyone,
> >
> > Please review and vote on the
release candidate #1 for the
> version
> > 2.7.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release
(please provide specific
> comments)
> >
> > The complete staging area is
available for your review, which
> includes:
> > * JIRA release notes [1],
> > * the official Apache source release
to be deployed to
> dist.apache.org <http://dist.apache.org/>
<http://dist.apache.org <http://dist.apache.org/>>
> > <http://dist.apache.org
<http://dist.apache.org/>> [2], which is signed
with the key with
> > fingerprint 45C60AAAD115F560 [3],
> > * all artifacts to be deployed to
the Maven Central
> Repository [4],
> > * source code tag "v2.7.0-RC1" [5],
> > * website pull request listing the
release and publishing the API
> > reference manual [6].
> > * Java artifacts were built with
Gradle 4.8 and OpenJDK
> > 1.8.0_181-8u181-b13-1~deb9u1-b13.
> > * Python artifacts are deployed
along with the source release
> to the
> > dist.apache.org
<http://dist.apache.org/> <http://dist.apache.org
<http://dist.apache.org/>>
> <http://dist.apache.org
<http://dist.apache.org/>> [2].
> >
> > The vote will be open for at least
72 hours. It is adopted by
> majority
> > approval, with at least 3 PMC
affirmative votes.
> >
> > Thanks,
> > Charles
> >
> > [1]
> >
>
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12343654
> > [2]
https://dist.apache.org/repos/dist/dev/beam/2.7.0
> > [3]
https://dist.apache.org/repos/dist/dev/beam/KEYS
> > [4]
>
https://repository.apache.org/content/repositories/orgapachebeam-1046/
> > [5]
https://github.com/apache/beam/tree/v2.7.0-RC1
> > [6]
https://github.com/apache/beam-site/pull/549
>
> --
> Jean-Baptiste Onofré
> jbono...@apache.org
<mailto:jbono...@apache.org>
<mailto:jbono...@apache.org
<mailto:jbono...@apache.org>>
> http://blog.nanthrax.net
<http://blog.nanthrax.net/>
> Talend - http://www.talend.com
<http://www.talend.com/>
>
