Re: [VOTE] Release 2.29.0, release candidate #1
On Fri, Apr 16, 2021 at 4:02 AM Kenneth Knowles wrote: > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org [2], > which is signed with the key with fingerprint > 03DBA3E6ABDD04BFD1558DC16ED551A8AE02461C [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag "v2.29.0-RC1" [5], > * website pull request listing the release [6], publishing the API reference > manual [7], and the blog post [8]. > * Java artifacts were built with Maven MAVEN_VERSION and OpenJDK/Oracle JDK > JDK_VERSION. Are the MAVEN_VERSION and OpenJDK/Oracle JDK JDK_VERSION supposed to be filled in with numbers? -- Elliotte Rusty Harold elh...@ibiblio.org
Re: protoc issues in docker container
https://issues.apache.org/jira/browse/BEAM-12149?filter=-2 On Thu, Apr 15, 2021 at 3:39 PM Brian Hulette wrote: > > Hm I thought we tested out docker builds within the development container. > Maybe this only happens for certain docker setups. Could you file a jira for > this? > > I came across this GitHub issue [1] that cites a similar problem, but sadly > doesn't reveal any details about the root cause or a solution. > > Brian > > [1] https://github.com/google/protobuf-gradle-plugin/issues/165 > > On Fri, Apr 9, 2021 at 3:33 PM Evan Galpin wrote: >> >> I ran into this same issue recently as well. It looks like you may have >> already found the same, but I can say that adding execute permissions did >> fix the issue for me. >> >> If I recall, adding execute perms lead to another very similar issue where >> an additional exe required the same fix. Once applied to both, everything >> seemed to get along fine in my case. >> >> Thanks, >> Evan >> >> On Fri, Apr 9, 2021 at 18:28 Elliotte Rusty Harold >> wrote: >>> >>> I tried to run some tests using the Docker build environment (neat >>> idea by the way) and hit this: >>> >>> $ ./gradlew -p sdks/java/io/cassandra check >>> Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could >>> not be reused, use --status for details >>> Configuration on demand is an incubating feature. >>> > Task :model:pipeline:generateProto FAILED >>> >>> FAILURE: Build failed with an exception. >>> >>> * What went wrong: >>> Execution failed for task ':model:pipeline:generateProto'. >>> > java.io.IOException: Cannot run program >>> > "/home/elharo/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protoc/3.15.3/8fb78f9edc8192143d947edc2217aafaa5d5f79b/protoc-3.15.3-linux-x86_64.exe": >>> > error=13, Permission denied >>> >>> I'm not sure immediately what's wrong, but perhaps the Docker >>> container isn't configured quite right? There could be more config I >>> need to do, but I thought this would be set up out of the box in the >>> Docker image. I do see that protoc does not have execute permission: >>> >>> $ ls -l >>> /home/elharo/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protoc/3.15.3/8fb78f9edc8192143d947edc2217aafaa5d5f79b/ >>> total 5120 >>> -rw-r--r-- 1 elharo users 5241944 Apr 9 22:17 >>> protoc-3.15.3-linux-x86_64.exe >>> >>> >>> >>> -- >>> Elliotte Rusty Harold >>> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
protoc issues in docker container
I tried to run some tests using the Docker build environment (neat idea by the way) and hit this: $ ./gradlew -p sdks/java/io/cassandra check Starting a Gradle Daemon, 1 incompatible and 1 stopped Daemons could not be reused, use --status for details Configuration on demand is an incubating feature. > Task :model:pipeline:generateProto FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':model:pipeline:generateProto'. > java.io.IOException: Cannot run program > "/home/elharo/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protoc/3.15.3/8fb78f9edc8192143d947edc2217aafaa5d5f79b/protoc-3.15.3-linux-x86_64.exe": > error=13, Permission denied I'm not sure immediately what's wrong, but perhaps the Docker container isn't configured quite right? There could be more config I need to do, but I thought this would be set up out of the box in the Docker image. I do see that protoc does not have execute permission: $ ls -l /home/elharo/.gradle/caches/modules-2/files-2.1/com.google.protobuf/protoc/3.15.3/8fb78f9edc8192143d947edc2217aafaa5d5f79b/ total 5120 -rw-r--r-- 1 elharo users 5241944 Apr 9 22:17 protoc-3.15.3-linux-x86_64.exe -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Long term support versions of Beam Java
On Tue, Apr 6, 2021 at 9:43 PM Jan Lukavský wrote: > > Hi, > do we know what is the reason users stay on an older version of Beam? My > guess would be that it is not related to API changes, but more likely to > state incompatibility. Maybe if we could figure out a way which would enable > a smooth migration of state (and timers) between Beam versions, that might > help? The migration would probably have to be runner-dependent, but Beam > might offer some tooling to make this easier. One example would be coder > evolution, where we currently do not have the option of "reading old way, > writing new way" with some "coder-version-registry". I suppose there might > have been a discussion about this in the past, does anyone know of any > conclusion? > Different customers have different requirements. The projects and teams I'm talking to simply don't want to invest engineering time in migrating as often as they have to now. They want to be able to ship a working product and not worry about it for a year or more. They recognize the possibility of unforeseen bugs and security issues that might require an update. However they'd like that update to be minimal and not require them to update their own code. Nor do they want to run the risk of performance regressions and newly introduced bugs and security issues. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Long term support versions of Beam Java
On Tue, Apr 6, 2021 at 8:55 PM Robert Bradshaw wrote: > > I do think there's value in having an LTS release, if there's sufficient > interest to fund it (specifically, figuring out who would be backporting > fixes and cutting the new releases). > I am reasonably confident that Google is willing to contribute the necessary engineering hours to support this. -- Elliotte Rusty Harold elh...@ibiblio.org
Long term support versions of Beam Java
Hi, I'd like to return to the discussion around a long term support release that first came up here in 2018: https://lists.apache.org/thread.html/6ec572d8edfe93225edebec18792cbcf44ef447ffe54ea35549cdafe%40%3Cdev.beam.apache.org%3E This is important to some Google Cloud Dataflow Java customers, and likely others as well. Specifically, I'd like to propose cutting an LTS release off a branch and maintaining it with critical bug fixes and security updates for 18 months. Right now we're finding that the current one year support period and six week release cycle is a tad fast for some customers. There's some wiggle room in terms of what's "critical", but in that category I include security fixes and data integrity issues. Essentially, this is any bug so bad that, if found in a new release, we'd recommend customers wait for the fix before upgrading to the latest and greatest. The difference is we'd backport the patch to the not-latest-and-greatest release. To run something up the flagpole, I propose: 1. 2.28.0 becomes the first LTS release. 2. New patch versions are released as 2.28.1, 2.28.2, etc. 3. Patch releases do not change API, at all, except in the unlikely event this is absolutely required for a security fix. 4. Dependencies are not upgraded in patch releases unless required to fix a critical bug or security issue. 5. In a year, cut a new LTS release from whatever is then current so there's some overlap to give customers time to switch over. I picked 2.28.0 since it's the most recent release, and I prefer to stay off the bleeding edge for longterm support. This would also enable customers to develop on top of it sooner. However I understand others may well prefer to pick a different release such as 2.29.0 or 2.30.0. I'm OK with whatever recent version the community picks. Thoughts? -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Builds Meeting this Thursday
On Mon, Jan 18, 2021 at 10:49 AM Ismaël Mejía wrote: > > Thanks for sharing this Pablo, This looks super interesting. We should > see if it could make sense to migrate our Jenkins infra to GitHub > Actions given that it is free and quickly becoming the new 'standard', > Good points it is 'free' because we will bring our machines and Google > pays :) bad points we will become 100% github dependant. > Github actions have a really big advantage over Jenkins: they run on forks, not just branches. This is very useful to non-commmiter contributors. On the minus side it's not clear if one can see the logs from the integration tests, which is blocking some work in the maven-site-plugin: https://github.com/apache/maven-site-plugin/pull/34#issuecomment-762207488 -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Contributing Twister2 runner to Apache Beam
Generic question without commenting on Twister2 specifically: Should runners, current and future, be in the same repository as Beam core? Can or should they be completely separate products with their own release cycles? Generally, loose coupling leads to more maintainable, reliable projects. Specifically, Cassandra is holding back some other changes in Beam and I really wish it didn't have the ability to block progress. The more different runners we have in core, the worse this problem is likely to become. On Wed, Mar 4, 2020 at 2:03 PM Pulasthi Supun Wickramasinghe wrote: > > Hi > > I believe the pull request is pretty complete now with the help of Ismaël. > Kenn, would you be able to take a look at it and suggest any changes if > needed?. The build checks and validations tests are passing at the moment. I > will start working on the documentation that you mentioned in an earlier > email separately. > > Best Regards, > Pulasthi > > > > > > On Tue, Feb 18, 2020 at 1:45 PM Pulasthi Supun Wickramasinghe > wrote: >> >> Hi All, >> >> I have created the initial pull request [1] to contribute the Twister2 Beam >> runner to the Apache Beam codebase. More information on Twister2 can be >> found here[2] and the Twister2 codebase is available here[3]. At the moment >> only batch mode is supported in the runner, but we are planning to add >> stream support and implement a portable runner for Twister2 in the near >> future. >> >> As Kenn pointed out in an earlier email it would be great to have inputs >> from the community regarding this contribution since it is a sizable one. I >> am sure there are many improvements that can be done in the contributed >> codebase with input from the community. >> >> [1] https://github.com/apache/beam/pull/10888 >> [2] https://twister2.org/ >> [3] https://github.com/DSC-SPIDAL/twister2 >> >> Best Regards, >> Pulasthi >> -- >> Pulasthi S. Wickramasinghe >> PhD Candidate | Research Assistant >> School of Informatics and Computing | Digital Science Center >> Indiana University, Bloomington >> cell: 224-386-9035 > > > > -- > Pulasthi S. Wickramasinghe > PhD Candidate | Research Assistant > School of Informatics and Computing | Digital Science Center > Indiana University, Bloomington > cell: 224-386-9035 -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
I'm not saying it is a best practice. I'm suggesting we should figure
out whether it's a best practice, worst practice, or somewhere in
between and document in which situations it is or is not a good idea.
Up till now I haven't really considered it an alternative to shading
so it seems worth thinking about more deeply.
On Fri, Jan 17, 2020 at 1:35 PM Kenneth Knowles wrote:
>
>
>
> On Fri, Jan 17, 2020 at 10:21 AM Elliotte Rusty Harold
> wrote:
>>
>> Thanks. That's a very clear explanation. I should think about how to
>> incorporate that into https://jlbp.dev
>
>
> Might want to give it some time before calling it a best practice. Maybe
> we'll come to regret it :-). But I'm willing to say it is a better
> alternative to shading in most cases, with the major exception being
> generated code (like proto libs) that depend on libraries (like
> protobuf-java). Ideally, proto compiler would vendor the dep into each
> generated lib so we don't have to deal with the issue at all. Instead we do
> it ourselves.
>
> Kenn
>
>>
>>
>>
>> On Fri, Jan 17, 2020 at 12:12 PM Aaron Dixon wrote:
>> >
>> > @Tomo Suzuki Thanks for looking at this and your thorough analysis// let
>> > me know if I can help with any regression testing, CRs, or more context
>> > anything else.
>> >
>> > ---
>> >
>> > @Elliotte I was a little confused at first re: vendoring vs shading but
>> > here is how I understand it (happy to be corrected if anything I say here
>> > is a mischaracterization); I tend to over-explain for my own clarity
>> > fwiw--using Guava as an example:
>> >
>> > With SHADING you have your project (e.g., Beam) depend directly on Guava
>> > and its public package names. Your build will then compile and link to
>> > these package names and only after-the-fact [1] rewrite the byte-code to
>> > rename the guava package. This entails both byte-code rewriting the Guava
>> > class files (to rename the packages of the Guava code itself) and
>> > byte-code rewriting all of your project class files that import Guava (to
>> > change the import to use the new package name).
>> >
>> > With VENDORING you release a renamed Guava ("vendor") JAR and depend on
>> > that directly from your project. So your project's source code itself
>> > imports the (repackated) vendor JAR and depends on the renamed packages
>> > directly. Your project build then has no need for shading or byte code
>> > rewriting of its own classes. I see vendoring making it very clear what is
>> > going on when looking at a project's source code and its dependencies, and
>> > there are likely other tangible brass-tacks benefits to vendoring besides
>> > the conceptual clarity over shading.
>> >
>> > [1] You can see that Maven shade:shade goal binds by default to Maven's
>> > package phase (which is well after after compilation).
>> >
>> > On Fri, Jan 17, 2020 at 9:16 AM Tomo Suzuki wrote:
>> >>
>> >> Ismaël and I think the short-term solution for Aaron's issue is to
>> >> define Beam's own class for Avro's TimestampConversion. Luckily it's
>> >> the only blocker for Avro 1.9 I found.
>> >> I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
>> >> PR https://github.com/apache/beam/pull/10628 .
>> >>
>> >> Regards,
>> >> Tomo
>> >>
>> >> On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
>> >> wrote:
>> >> >
>> >> > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote:
>> >> > >
>> >> > > Regarding Google's advice about shading: don't go to a "one version
>> >> > > rule" monorepo for advice about solving diamond dependencies in the
>> >> > > wild.
>> >> >
>> >> > FWIW these docs came out of the open source, Github, multirepo, many
>> >> > versions part of Google Cloud, not the mono repo world of google3.
>> >> > Though I will say the more time I spend trying to deal with these
>> >> > issues the more I love the monorepo.
>> >> >
>> >> > > It is a useful description of the pitfalls. We (and Flink before us,
>> >> > > and likely many more) are already doing something that avoids many of
>> >> > > them or makes them less likely. Building a separate vendored library
>> >> > > is simpler and more robust than the "shade during build".
>> >> > >
>> >> >
>> >> > I've only encountered "vendoring" as distinct from shading in Beam.
>> >> > Are there more details about this anywhere? A quick Google search only
>> >> > turned up one Beam doc that pointed to Go docs and a cloud endpoints
>> >> > doc that seemed to use it as a synonym for shading.
>> >> >
>> >> >
>> >> > --
>> >> > Elliotte Rusty Harold
>> >> > elh...@ibiblio.org
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Tomo
>>
>>
>>
>> --
>> Elliotte Rusty Harold
>> elh...@ibiblio.org
--
Elliotte Rusty Harold
elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
Thanks. That's a very clear explanation. I should think about how to
incorporate that into https://jlbp.dev
On Fri, Jan 17, 2020 at 12:12 PM Aaron Dixon wrote:
>
> @Tomo Suzuki Thanks for looking at this and your thorough analysis// let me
> know if I can help with any regression testing, CRs, or more context anything
> else.
>
> ---
>
> @Elliotte I was a little confused at first re: vendoring vs shading but here
> is how I understand it (happy to be corrected if anything I say here is a
> mischaracterization); I tend to over-explain for my own clarity fwiw--using
> Guava as an example:
>
> With SHADING you have your project (e.g., Beam) depend directly on Guava and
> its public package names. Your build will then compile and link to these
> package names and only after-the-fact [1] rewrite the byte-code to rename the
> guava package. This entails both byte-code rewriting the Guava class files
> (to rename the packages of the Guava code itself) and byte-code rewriting all
> of your project class files that import Guava (to change the import to use
> the new package name).
>
> With VENDORING you release a renamed Guava ("vendor") JAR and depend on that
> directly from your project. So your project's source code itself imports the
> (repackated) vendor JAR and depends on the renamed packages directly. Your
> project build then has no need for shading or byte code rewriting of its own
> classes. I see vendoring making it very clear what is going on when looking
> at a project's source code and its dependencies, and there are likely other
> tangible brass-tacks benefits to vendoring besides the conceptual clarity
> over shading.
>
> [1] You can see that Maven shade:shade goal binds by default to Maven's
> package phase (which is well after after compilation).
>
> On Fri, Jan 17, 2020 at 9:16 AM Tomo Suzuki wrote:
>>
>> Ismaël and I think the short-term solution for Aaron's issue is to
>> define Beam's own class for Avro's TimestampConversion. Luckily it's
>> the only blocker for Avro 1.9 I found.
>> I created a ticket https://issues.apache.org/jira/browse/BEAM-9144 and
>> PR https://github.com/apache/beam/pull/10628 .
>>
>> Regards,
>> Tomo
>>
>> On Fri, Jan 17, 2020 at 7:32 AM Elliotte Rusty Harold
>> wrote:
>> >
>> > On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote:
>> > >
>> > > Regarding Google's advice about shading: don't go to a "one version
>> > > rule" monorepo for advice about solving diamond dependencies in the wild.
>> >
>> > FWIW these docs came out of the open source, Github, multirepo, many
>> > versions part of Google Cloud, not the mono repo world of google3.
>> > Though I will say the more time I spend trying to deal with these
>> > issues the more I love the monorepo.
>> >
>> > > It is a useful description of the pitfalls. We (and Flink before us, and
>> > > likely many more) are already doing something that avoids many of them
>> > > or makes them less likely. Building a separate vendored library is
>> > > simpler and more robust than the "shade during build".
>> > >
>> >
>> > I've only encountered "vendoring" as distinct from shading in Beam.
>> > Are there more details about this anywhere? A quick Google search only
>> > turned up one Beam doc that pointed to Go docs and a cloud endpoints
>> > doc that seemed to use it as a synonym for shading.
>> >
>> >
>> > --
>> > Elliotte Rusty Harold
>> > elh...@ibiblio.org
>>
>>
>>
>> --
>> Regards,
>> Tomo
--
Elliotte Rusty Harold
elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
On Thu, Jan 16, 2020 at 7:08 PM Kenneth Knowles wrote: > > Regarding Google's advice about shading: don't go to a "one version rule" > monorepo for advice about solving diamond dependencies in the wild. FWIW these docs came out of the open source, Github, multirepo, many versions part of Google Cloud, not the mono repo world of google3. Though I will say the more time I spend trying to deal with these issues the more I love the monorepo. > It is a useful description of the pitfalls. We (and Flink before us, and > likely many more) are already doing something that avoids many of them or > makes them less likely. Building a separate vendored library is simpler and > more robust than the "shade during build". > I've only encountered "vendoring" as distinct from shading in Beam. Are there more details about this anywhere? A quick Google search only turned up one Beam doc that pointed to Go docs and a cloud endpoints doc that seemed to use it as a synonym for shading. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
Avro does not follow semver. They update the major version when the serialization format changes and the minor version when the API changes in a backwards incompatible way. See https://issues.apache.org/jira/browse/AVRO-2687 On Thu, Jan 16, 2020 at 12:50 PM Luke Cwik wrote: > > Does avro not follow semantic versioning and upgrading to 1.9 should have > been backwards compatible or does our usage reach into the internals of avro? > > On Thu, Jan 16, 2020 at 6:16 AM Ismaël Mejía wrote: >> >> I forgot to explain why the most obvious path (just upgrade Avro to version >> 1.9.x) is not a valid long term solution. Other systems Beam runs on top of >> (e.g. Spark!) also leak Avro into their core so in the moment Spark moves up >> to Avro 1.9.x Spark runner users will be in a really fragile position where >> things will work until they don't (similar to Aaron's case) so a stronger >> reason >> to getAvro out of Beam core. >> >> >> On Thu, Jan 16, 2020 at 1:59 PM Elliotte Rusty Harold >> wrote: >>> >>> Shading should be a last resort: >>> >>> https://jlbp.dev/JLBP-18.html >>> >>> It tends to cause more problems than it solves. At best it's a stopgap >>> measure when you don't have the resources to fix the real problem. In >>> this case it sounds like the real issue is that AVRO is not stable. >>> There are at least three other solutions in a case like this: >>> >>> 1. Fix Avro at the root. >>> 2. Fork Avro and then fix it. >>> 3. Stop depending on Avro. >>> >>> None of these are trivial which is why shading gets considered. >>> However shading doesn't fix the underlying problems, and ultimately >>> makes a product as unreliable as its least reliable dependency. :-( >>> >>> On Thu, Jan 16, 2020 at 2:01 AM jincheng sun >>> wrote: >>> > >>> > I found that there are several dependencies shaded and planned to made as >>> > vendored artifacts in [1]. I'm not sure why Avro is not shaded before. >>> > From my point of view, it's a good idea to shade Avro and make it a >>> > vendored artifact if there are no special reasons blocking us to do that. >>> > Regarding to how to create a vendored artifact, you can refer to [2] for >>> > more details. >>> > >>> > Best, >>> > Jincheng >>> > >>> > [1] https://issues.apache.org/jira/browse/BEAM-5819 >>> > [2] https://github.com/apache/beam/blob/master/vendor/README.md >>> > >>> > >>> > Tomo Suzuki 于2020年1月16日周四 下午1:18写道： >>> >> >>> >> I've been upgrading dependencies around gRPC. This Avro-problem is >>> >> interesting to me. >>> >> I'll study BEAM-8388 more tomorrow. >>> >> >>> >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik wrote: >>> >> > >>> >> > +Tomo Suzuki +jincheng sun >>> >> > There have been a few contributors upgrading the dependencies and >>> >> > validating things not breaking by running the majority of the post >>> >> > commit integration tests and also using the linkage checker to show >>> >> > that we aren't worse off with respect to our dependency tree. Reaching >>> >> > out to them to help your is your best bet of getting these upgrades >>> >> > through. >>> >> > >>> >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon wrote: >>> >> >> >>> >> >> I meant to mention that we must use Avro 1.9.x as we rely on some >>> >> >> schema resolution fixes not present in 1.8.x - so am indeed blocked. >>> >> >> >>> >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: >>> >> >>> >>> >> >>> It looks like Avro version dependency from Beam has come up in the >>> >> >>> past [1, 2]. >>> >> >>> >>> >> >>> I'm currently on Beam 2.16.0, which has been compatible with my >>> >> >>> usage of Avro 1.9.x. >>> >> >>> >>> >> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 >>> >> >>> has some dependencies on Avro classes only available in 1.8.x. >>> >> >>> >>> >> >>> Wondering if anyone else is similar blocked and what it would take >>> >> >>> to prioritize Beam upgrading to 1.9.x or better using a shaded >>> >> >>> version so that clients can use their own Avro version for their own >>> >> >>> coding purposes. (Eg, I parse Avro messages from a KafkaIO source >>> >> >>> and need 1.9.x for this but am perfectly happy if Beam's Avro coding >>> >> >>> facilities used a shaded other version.) >>> >> >>> >>> >> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling >>> >> >>> community for discussion. >>> >> >>> >>> >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 >>> >> >>> [2] https://github.com/apache/beam/pull/9779 >>> >> >>> >>> >> >>> >> >>> >> -- >>> >> Regards, >>> >> Tomo >>> >>> >>> >>> -- >>> Elliotte Rusty Harold >>> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Beam's Avro 1.8.x dependency
Shading should be a last resort: https://jlbp.dev/JLBP-18.html It tends to cause more problems than it solves. At best it's a stopgap measure when you don't have the resources to fix the real problem. In this case it sounds like the real issue is that AVRO is not stable. There are at least three other solutions in a case like this: 1. Fix Avro at the root. 2. Fork Avro and then fix it. 3. Stop depending on Avro. None of these are trivial which is why shading gets considered. However shading doesn't fix the underlying problems, and ultimately makes a product as unreliable as its least reliable dependency. :-( On Thu, Jan 16, 2020 at 2:01 AM jincheng sun wrote: > > I found that there are several dependencies shaded and planned to made as > vendored artifacts in [1]. I'm not sure why Avro is not shaded before. From > my point of view, it's a good idea to shade Avro and make it a vendored > artifact if there are no special reasons blocking us to do that. Regarding to > how to create a vendored artifact, you can refer to [2] for more details. > > Best, > Jincheng > > [1] https://issues.apache.org/jira/browse/BEAM-5819 > [2] https://github.com/apache/beam/blob/master/vendor/README.md > > > Tomo Suzuki 于2020年1月16日周四 下午1:18写道： >> >> I've been upgrading dependencies around gRPC. This Avro-problem is >> interesting to me. >> I'll study BEAM-8388 more tomorrow. >> >> On Wed, Jan 15, 2020 at 10:51 PM Luke Cwik wrote: >> > >> > +Tomo Suzuki +jincheng sun >> > There have been a few contributors upgrading the dependencies and >> > validating things not breaking by running the majority of the post commit >> > integration tests and also using the linkage checker to show that we >> > aren't worse off with respect to our dependency tree. Reaching out to them >> > to help your is your best bet of getting these upgrades through. >> > >> > On Wed, Jan 15, 2020 at 6:52 PM Aaron Dixon wrote: >> >> >> >> I meant to mention that we must use Avro 1.9.x as we rely on some schema >> >> resolution fixes not present in 1.8.x - so am indeed blocked. >> >> >> >> On Wed, Jan 15, 2020 at 8:50 PM Aaron Dixon wrote: >> >>> >> >>> It looks like Avro version dependency from Beam has come up in the past >> >>> [1, 2]. >> >>> >> >>> I'm currently on Beam 2.16.0, which has been compatible with my usage of >> >>> Avro 1.9.x. >> >>> >> >>> But upgrading to Beam 2.17.0 is not possible for us now that 2.17.0 has >> >>> some dependencies on Avro classes only available in 1.8.x. >> >>> >> >>> Wondering if anyone else is similar blocked and what it would take to >> >>> prioritize Beam upgrading to 1.9.x or better using a shaded version so >> >>> that clients can use their own Avro version for their own coding >> >>> purposes. (Eg, I parse Avro messages from a KafkaIO source and need >> >>> 1.9.x for this but am perfectly happy if Beam's Avro coding facilities >> >>> used a shaded other version.) >> >>> >> >>> I've made a comment on BEAM-8388 [1] to this effect. But polling >> >>> community for discussion. >> >>> >> >>> [1] https://issues.apache.org/jira/browse/BEAM-8388 >> >>> [2] https://github.com/apache/beam/pull/9779 >> >>> >> >> >> -- >> Regards, >> Tomo -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Unifying Build/contributing instructions
That's two votes for https://beam.apache.org/contribute/contribution-guide/ and a lot of abstentions. I'll update the PR to move content to https://beam.apache.org/contribute/contribution-guide/ On Thu, Dec 19, 2019 at 12:29 PM Luke Cwik wrote: > > +1 on Kenn's suggestion. > > On Thu, Dec 12, 2019 at 8:17 PM Kenneth Knowles wrote: >> >> Thanks for taking this on! My preference would be to have CONTRIBUTING.md >> link to https://beam.apache.org/contribute/contribution-guide/ and focus >> work on the latter. >> >> Kenn >> >> On Thu, Dec 12, 2019 at 12:38 PM Elliotte Rusty Harold >> wrote: >>> >>> I've started work on updating and combine the four (or omre?) >>> different pages where build instructions are found. The initial PR is >>> here: >>> >>> https://github.com/apache/beam/pull/10366 >>> >>> To put a stake in the ground, this PR chooses CONTRIBUTING.md as the >>> ultimate source of truth. A possible alternative is to unify around >>> https://beam.apache.org/contribute/contribution-guide/ >>> >>> I'm not wedded to one or the other, but I do think we should pick one >>> and stick with it. If the community prefers to focus on >>> https://beam.apache.org/contribute/contribution-guide/ we can use that >>> instead. >>> >>> I've added some additional prerequisites to the instructions that were >>> not yet included. I don't have it all yet though. Any further >>> additions would be much appreciated. >>> >>> Please leave comments on the PR. >>> >>> -- >>> Elliotte Rusty Harold >>> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Unifying Build/contributing instructions
I've started work on updating and combine the four (or omre?) different pages where build instructions are found. The initial PR is here: https://github.com/apache/beam/pull/10366 To put a stake in the ground, this PR chooses CONTRIBUTING.md as the ultimate source of truth. A possible alternative is to unify around https://beam.apache.org/contribute/contribution-guide/ I'm not wedded to one or the other, but I do think we should pick one and stick with it. If the community prefers to focus on https://beam.apache.org/contribute/contribution-guide/ we can use that instead. I've added some additional prerequisites to the instructions that were not yet included. I don't have it all yet though. Any further additions would be much appreciated. Please leave comments on the PR. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: [DISCUSS] @Experimental annotations - processes and alternatives
On Wed, Nov 27, 2019 at 1:12 PM Kenneth Knowles wrote: > > *Opt-in*: This is a powerful idea that I think changes everything. >- for an experimental new IO, a separate artifact; this way we can also > see downloads >- for experimental code fragments, add checkState that the relevant > experiment is turned on via flags To be clear the experimental artifact would have the same group ID and artifact ID but a different version than the non-experimental artifacts? E.g. org.apache.beam:beam-runners-gcp-gcemd:2.4.0-experimental That could work. Changing the artifact ID or the package name would risk split package issues and diamond dependency problems. We'd still need to be careful about mixing experimental and non-experimental artifacts. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: goVet and clickHouse tests failing
I did get through this one, and made the classic mistake of not immediately committing the steps I took to writing. I believe it involved some combination of setting go paths in environment variables. I seem to have added this to the end of my .profile: export GOROOT=/usr/local/go export GOPATH=$HOME/go/packages export PATH=$GOPATH/bin:$GOROOT/bin:$PATH According to my history, I also ran go get github.com/linkedin/goavro I really need to dedicate some time to cleaning up the build documentation. See https://issues.apache.org/jira/browse/BEAM-8798 Right now we have overlapping and sometimes contradictory docs in four different places: README.md CONTRIBUTING.md https://cwiki.apache.org/confluence/display/BEAM/Contributor+FAQ https://beam.apache.org/contribute/ We should probably pick one as the source of truth and rewrite the other three to simply point to it. I propose putting all checkout, build, test, commit, and push instructions in CONTRIBUTING.md in the repo. What do folks think? On Wed, Nov 27, 2019 at 3:51 AM Amogh Tiwari wrote: > > Hi Elliotte, > I am facing a similar goVet issue. It would be great if you can guide me > through the solution. Please let me know the steps that you followed. > Regards, > Amogh > > On Thu, Nov 21, 2019 at 6:06 PM Elliotte Rusty Harold > wrote: >> >> Tentatively, the goVet issue does seem to have been an issue with my >> Go install I have now cleaned up. The clickhouse issue remains, as do >> several others I'm working through. >> >> I've filed https://issues.apache.org/jira/browse/BEAM-8798 to >> consolidate and update the instructions for getting to a working >> build. >> >> On Thu, Nov 21, 2019 at 6:10 AM Elliotte Rusty Harold >> wrote: >> > >> > I'm slowly working my way through getting the tests to run and pass. >> > We have a lot of work to do on the contributing docs to explain how to >> > setup and run the build. There's clearly a lot of knowledge in >> > developers' heads and workstations that hasn't yet made it into the >> > docs. >> > >> > The latest is a problem finding "github.com/linkedin/goavro" when I >> > run goVet. I'm not a go person. Is this something that requires an >> > extra install? If so, how is it installed? Or is this some error in >> > the build.gradles? Or perhaps my go config is borked and gradle is >> > looking in the wrong directory? >> > >> > > Task :sdks:go:examples:resolveBuildDependencies >> > Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go >> > .gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2: >> > cannot find package "github.com/linkedin/goavro" in any of: >> > >> > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro >> > (vendor tree) >> > >> > /home/elharo/.gradle/go/binary/1.12/go/src/github.com/linkedin/goavro >> > (from $GOROOT) >> > >> > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/linkedin/goavro >> > (from $GOPATH) >> > >> > > Task :sdks:go:examples:goVet FAILED >> > >> > I'm also seeing failures in ClickHouseIOTest: >> > >> > > Task :sdks:java:io:clickhouse:test >> > >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED >> > java.lang.IllegalStateException >> > >> > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED >> > java.lang.NullPointerException >> > >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED >> > java.lang.IllegalStateException >> > >> > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED >> > java.lang.NullPointerException >> > >> > -- >> > Elliotte Rusty Harold >> > elh...@ibiblio.org >> >> >> >> -- >> Elliotte Rusty Harold >> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: goVet and clickHouse tests failing
That would explain it. I didn't have Docker installed. I do now, so let's see if it passes. Ultimately the root cause for a lot of this seems to be incomplete and conflicting docs. I hope we can invest some effort into improving this. On Thu, Nov 21, 2019 at 6:27 AM Gleb Kanterov wrote: > > :sdks:java:io:clickhouse:test is using testcontainers. Testcontainers is a > Java library that supports JUnit tests, providing lightweight, throwaway > instances of common databases, Selenium web browsers, or anything else that > can run in a Docker container. Therefore, ClickHouse tests require a local > Docker environment. It might make sense to move tests using testcontainers > into a separate gradle task if it's a common issue. > > On Thu, Nov 21, 2019 at 12:10 PM Elliotte Rusty Harold > wrote: >> >> I'm slowly working my way through getting the tests to run and pass. >> We have a lot of work to do on the contributing docs to explain how to >> setup and run the build. There's clearly a lot of knowledge in >> developers' heads and workstations that hasn't yet made it into the >> docs. >> >> The latest is a problem finding "github.com/linkedin/goavro" when I >> run goVet. I'm not a go person. Is this something that requires an >> extra install? If so, how is it installed? Or is this some error in >> the build.gradles? Or perhaps my go config is borked and gradle is >> looking in the wrong directory? >> >> > Task :sdks:go:examples:resolveBuildDependencies >> Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go >> .gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2: >> cannot find package "github.com/linkedin/goavro" in any of: >> >> /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro >> (vendor tree) >> /home/elharo/.gradle/go/binary/1.12/go/src/github.com/linkedin/goavro >> (from $GOROOT) >> >> /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/linkedin/goavro >> (from $GOPATH) >> >> > Task :sdks:go:examples:goVet FAILED >> >> I'm also seeing failures in ClickHouseIOTest: >> >> > Task :sdks:java:io:clickhouse:test >> >> org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED >> java.lang.IllegalStateException >> >> org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED >> java.lang.NullPointerException >> >> org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED >> java.lang.IllegalStateException >> >> org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED >> java.lang.NullPointerException >> >> -- >> Elliotte Rusty Harold >> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: goVet and clickHouse tests failing
Tentatively, the goVet issue does seem to have been an issue with my Go install I have now cleaned up. The clickhouse issue remains, as do several others I'm working through. I've filed https://issues.apache.org/jira/browse/BEAM-8798 to consolidate and update the instructions for getting to a working build. On Thu, Nov 21, 2019 at 6:10 AM Elliotte Rusty Harold wrote: > > I'm slowly working my way through getting the tests to run and pass. > We have a lot of work to do on the contributing docs to explain how to > setup and run the build. There's clearly a lot of knowledge in > developers' heads and workstations that hasn't yet made it into the > docs. > > The latest is a problem finding "github.com/linkedin/goavro" when I > run goVet. I'm not a go person. Is this something that requires an > extra install? If so, how is it installed? Or is this some error in > the build.gradles? Or perhaps my go config is borked and gradle is > looking in the wrong directory? > > > Task :sdks:go:examples:resolveBuildDependencies > Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go > .gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2: > cannot find package "github.com/linkedin/goavro" in any of: > > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro > (vendor tree) > /home/elharo/.gradle/go/binary/1.12/go/src/github.com/linkedin/goavro > (from $GOROOT) > > /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/linkedin/goavro > (from $GOPATH) > > > Task :sdks:go:examples:goVet FAILED > > I'm also seeing failures in ClickHouseIOTest: > > > Task :sdks:java:io:clickhouse:test > > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED > java.lang.IllegalStateException > > org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED > java.lang.NullPointerException > > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED > java.lang.IllegalStateException > > org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED > java.lang.NullPointerException > > -- > Elliotte Rusty Harold > elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
goVet and clickHouse tests failing
I'm slowly working my way through getting the tests to run and pass. We have a lot of work to do on the contributing docs to explain how to setup and run the build. There's clearly a lot of knowledge in developers' heads and workstations that hasn't yet made it into the docs. The latest is a problem finding "github.com/linkedin/goavro" when I run goVet. I'm not a go person. Is this something that requires an extra install? If so, how is it installed? Or is this some error in the build.gradles? Or perhaps my go config is borked and gradle is looking in the wrong directory? > Task :sdks:go:examples:resolveBuildDependencies Resolving ./github.com/apache/beam/sdks/go@/home/elharo/beam/sdks/go .gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/apache/beam/sdks/go/pkg/beam/io/avroio/avroio.go:28:2: cannot find package "github.com/linkedin/goavro" in any of: /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/apache/beam/sdks/go/examples/vendor/github.com/linkedin/goavro (vendor tree) /home/elharo/.gradle/go/binary/1.12/go/src/github.com/linkedin/goavro (from $GOROOT) /home/elharo/beam/sdks/go/examples/.gogradle/project_gopath/src/github.com/linkedin/goavro (from $GOPATH) > Task :sdks:go:examples:goVet FAILED I'm also seeing failures in ClickHouseIOTest: > Task :sdks:java:io:clickhouse:test org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED java.lang.IllegalStateException org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED java.lang.NullPointerException org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED java.lang.IllegalStateException org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED java.lang.NullPointerException -- Elliotte Rusty Harold elh...@ibiblio.org
Re: GCP libraries up-to-date versions in Java
On Wed, Nov 20, 2019 at 1:43 PM Luke Cwik wrote: > > Minor note that Gradle 5 added support for BOMs[1]. > > I think attempting to perform the upgrade (whether to use BOM or not) will be > a concerted effort every time to minimize the amount of breakage to users > while maximizing compatibility with the OSS ecosystem. Unfortunately I'm not > aware of any dependency analysis tooling that can perform some validation > stating that something is safe or not. If such a tool existed, it would make > it much easier for projects to perform upgrades and would also help users as > well. Funny you should ask. We've been working on tooling like that such as the linkage monitor and the Maven enforcer rule: https://github.com/GoogleCloudPlatform/cloud-opensource-java Mostly Maven based for the time being, but updates are possible. At the end of the day, though, this is no substitute for extensive unit and integration test suites. If we don't have those, then dependencies are the least of our worries. If we do have those, we can move forward in reasonable confidence that minor version upgrades won't break anything without causing a test to fail. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: GCP libraries up-to-date versions in Java
BOM or no BOM is an implementation detail. Using com.google.clou:libraries-bom would make dependency management simpler for developers, but the real issue is whether Beam can continue to work with very old versions of the many libraries it depends on. Even if this is acceptable for Beam, it's unlikely to be feasible for anyone who needs to mix Beam code with other code. There should be no self-incompatibility between Google minor version releases. All the Google libraries in question follow semantic versioning. E.g. Pubsub 1.43 would be fully API compatible with Pubsub 1.28, though not the reverse. However there are likely to be important bug fixes in 1.43 and definitely new features that 1.28 would not have. If there are any edge cases where this is not true, that's a bug and if you file it against the repo we'll try to fix it. We're also installing tooling to make this less likely to happen by accident. However, right now any such problem is rare. Behavior differences are another story. It is entirely possible that something like Pubsub 1.28 would simply no longer function due to changes at the backend. There's a deprecation cycle, announcements, and transition periods in all such cases; but a project like Beam can't stay on old versions forever. If they try, the backends will shift out from under them. Looking at Beam's dependencies, the only case where there are major version changes to address is Guava. This will take some work, but not an excessive amount. We should be able to move this up to 28.1-android with few code changes and no further API breaking changes in that library are planned for the future. The remaining issues are pre-1.0 libraries. OpenCensus is a particular thorn in my side. Ideally these should not be used, at all. However if we must, we should not expose them on the Beam API surface and we need to move them forward quickly as they change. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Python tests and setup
and probably related to that: https://github.com/pyinstaller/pyinstaller/issues/4064 On Mon, Nov 18, 2019 at 1:47 PM Elliotte Rusty Harold wrote: > > Probably related: > > > > Task :sdks:python:test-suites:tox:py36:setupVirtualenv FAILED > Traceback (most recent call last): > File "/usr/lib/python3/dist-packages/virtualenv.py", line 25, in > import distutils.sysconfig > ModuleNotFoundError: No module named 'distutils.sysconfig' > Running virtualenv with interpreter /usr/bin/python3.6 > > On Mon, Nov 18, 2019 at 1:45 PM Elliotte Rusty Harold > wrote: > > > > My Python tests are now failing like so with a simple .gradlew check > > and a clean checkout: > > > > FAILURE: Build failed with an exception. > > > > * What went wrong: > > Execution failed for task > > ':sdks:python:test-suites:tox:py36:setupVirtualenv'. > > > Process 'command 'virtualenv'' finished with non-zero exit value 1 > > > > Any suggestions? > > > > The setup instructions on github do not appear to be up to date. See > > https://issues.apache.org/jira/browse/BEAM-8735 > > It wouldn't be a surprise if there are other missing steps to be followed. > > > > Related, is there a way to run all the Java tests and only the Java tests? > > > > -- > > Elliotte Rusty Harold > > elh...@ibiblio.org > > > > -- > Elliotte Rusty Harold > elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Python tests and setup
Probably related: > Task :sdks:python:test-suites:tox:py36:setupVirtualenv FAILED Traceback (most recent call last): File "/usr/lib/python3/dist-packages/virtualenv.py", line 25, in import distutils.sysconfig ModuleNotFoundError: No module named 'distutils.sysconfig' Running virtualenv with interpreter /usr/bin/python3.6 On Mon, Nov 18, 2019 at 1:45 PM Elliotte Rusty Harold wrote: > > My Python tests are now failing like so with a simple .gradlew check > and a clean checkout: > > FAILURE: Build failed with an exception. > > * What went wrong: > Execution failed for task ':sdks:python:test-suites:tox:py36:setupVirtualenv'. > > Process 'command 'virtualenv'' finished with non-zero exit value 1 > > Any suggestions? > > The setup instructions on github do not appear to be up to date. See > https://issues.apache.org/jira/browse/BEAM-8735 > It wouldn't be a surprise if there are other missing steps to be followed. > > Related, is there a way to run all the Java tests and only the Java tests? > > -- > Elliotte Rusty Harold > elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Python tests and setup
My Python tests are now failing like so with a simple .gradlew check and a clean checkout: FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':sdks:python:test-suites:tox:py36:setupVirtualenv'. > Process 'command 'virtualenv'' finished with non-zero exit value 1 Any suggestions? The setup instructions on github do not appear to be up to date. See https://issues.apache.org/jira/browse/BEAM-8735 It wouldn't be a surprise if there are other missing steps to be followed. Related, is there a way to run all the Java tests and only the Java tests? -- Elliotte Rusty Harold elh...@ibiblio.org
Wiki access
Hello, May I please have access to edit the Wiki? username is elharo Thanks. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: [Discuss] Beam Summit 2020 Dates & locations
The U.S. sadly is not a reliable destination for international conferences these days. Almost every conference I go to, big and small, has at least one speaker, sometimes more, who can't get into the country. Canada seems worth considering. Vancouver, Montreal, and Toronto are all convenient. On Wed, Nov 6, 2019 at 2:17 PM Griselda Cuevas wrote: > > Hi Beam Community! > > I'd like to kick off a thread to discuss potential dates and venues for the > 2020 Beam Summits. > > I did some research on industry conferences happening in 2020 and > pre-selected a few ranges as follows: > > (2 days) NA between mid-May and mid-June > (2 days) EU mid October > (1 day) Asia Mini Summit: March > > I'd like to hear your thoughts on these dates and get consensus on exact > dates as the convo progresses. > > For locations these are the options I reviewed: > > NA: Austin Texas, Berkeley California, Mexico City. > Europe: Warsaw, Barcelona, Paris > Asia: Singapore > > Let the discussion begin! > G (on behalf of the Beam Summit Steering Committee) > > > -- Elliotte Rusty Harold elh...@ibiblio.org
clickhouse tests failing
At head I noticed the following: $ ./gradlew -p sdks/java/io/ check Configuration on demand is an incubating feature. > Task :sdks:java:io:clickhouse:test org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED java.lang.IllegalStateException org.apache.beam.sdk.io.clickhouse.ClickHouseIOTest > classMethod FAILED java.lang.NullPointerException org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED java.lang.IllegalStateException org.apache.beam.sdk.io.clickhouse.AtomicInsertTest > classMethod FAILED java.lang.NullPointerException 29 tests completed, 4 failed > Task :sdks:java:io:clickhouse:test FAILED FAILURE: Build failed with an exception. Is anyone else seeing this? Are the tests expected to pass, or is there some requirement (e.g. Java 11) that I might be missing? -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Java 11 compatibility question
If somebody is using JPMS and they attempt to import beam, they get a compile time error. Some other projects I work on have been getting user reports about this. See https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-19.md for more details. On Tue, Aug 20, 2019 at 5:30 PM Ahmet Altay wrote: > > > > On Tue, Aug 20, 2019 at 8:37 AM Elliotte Rusty Harold > wrote: >> >> >> >> On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía wrote: >>> >>> a per case approach (the exception could be portable runners not based on >>> Java). >>> >>> Of course other definitions of being Java 11 compatible are interesting but >>> probably not part of our current scope. Actions like change the codebase to >>> use Java 11 specific APIs / idioms, publish Java 11 specific artifacts or >>> use Java Platform Modules (JPM). All of these may be nice to have but are >>> probably less important for end users who may just want to be able to use >>> Beam in its current form in Java 11 VMs. >>> >>> What do others think? Is this enough to announce Java 11 compatibility and >>> add the documentation to the webpage? >> >> >> No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really >> does need to be compatible with JPMS-using apps. The bare minimum here is >> avoiding split packages, and that needs to include all transitive >> dependencies, not just Beam itself. I don't think we meet that bar now. > > > For my understanding, what would be the limitations of Beam's dependencies > having split dependencies? Would it limit Beam users from using 3rd party > libraries that require JPMS supports? Would it be in scope for Beam to get > its dependencies to meet a certain bar? > > Ismaël's definition of being able to run Beam published dependencies in Java > 11 VM sounds enough to me "to announce Java 11 compatibility _for Beam_". > >> >> >> -- >> Elliotte Rusty Harold >> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Java 11 compatibility question
On Tue, Aug 20, 2019 at 7:51 AM Ismaël Mejía wrote: > a per case approach (the exception could be portable runners not based on > Java). > > Of course other definitions of being Java 11 compatible are interesting > but probably not part of our current scope. Actions like change the > codebase to use Java 11 specific APIs / idioms, publish Java 11 specific > artifacts or use Java Platform Modules (JPM). All of these may be nice to > have but are probably less important for end users who may just want to be > able to use Beam in its current form in Java 11 VMs. > > What do others think? Is this enough to announce Java 11 compatibility and > add the documentation to the webpage? > No, it isn't, I fear. We don't have to use JPMS in Beam, but Beam really does need to be compatible with JPMS-using apps. The bare minimum here is avoiding split packages, and that needs to include all transitive dependencies, not just Beam itself. I don't think we meet that bar now. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Java 11 compatibility question
Another useful thing to improve Java 11 support is to add Automatic-Module-Name headers to the various JARs Beam publishes: https://github.com/GoogleCloudPlatform/cloud-opensource-java/blob/master/library-best-practices/JLBP-20.md If a JAR doesn't include this, Java synthezizes one from the name of the jar file, and things get wonky fast. This is a low risk change that has no effect on non-modular and pre-Java-9 apps. On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia wrote: > Hi everyone, > > I want to gather the collective knowledge here about Java 11 compatibility > and ask about the tests needed to deem Beam compatible with JDK 11. > > Right now concerning testing JDK 11 compatibility I implemented: > >- Jenkins jobs for running ValidatesRunner test sets in both Direct >and Dataflow runners, [1], [2] >- ValidatesRunner portability API tests for Dataflow [3], >- examples in normal and portable mode for the Dataflow runner [4], >[5]. > > > Are these tests sufficient to say that we’re java 11 compatible? What > other aspects do we need to test to be able to say that? > > > Regards, > > > Michał > > [1] > https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Direct/ > [2] > https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_Dataflow/ > [3] > https://builds.apache.org/job/beam_PostCommit_Java11_ValidatesRunner_PortabilityApi_Dataflow/ > [4] > https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow/ > [5] > https://builds.apache.org/job/beam_PostCommit_Java11_Examples_Dataflow_Portability/ > > -- > > Michał Walenia > Polidea <https://www.polidea.com/> | Software Engineer > > M: +48 791 432 002 <+48791432002> > E: michal.wale...@polidea.com > > Unique Tech > Check out our projects! <https://www.polidea.com/our-work> > -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Java 11 compatibility question
gRPC bug here: https://github.com/grpc/grpc-java/issues/3522 google-cloud-java bug: https://github.com/googleapis/google-cloud-java/issues/5760 Neither has a cheap or easy fix, I'm afraid. Commenting on these issues might help us prove that there's a demand to priorotize these compared to other work. If anyone has a support contract and could file a ticket asking for a fix, that would help even more. Those are the two I know about. There might be others elsewhere in the dependency tree. On Wed, Aug 7, 2019 at 2:25 PM Lukasz Cwik wrote: > > Since java8 -> java11 is similar to python2 -> python3 migration, what was > the acceptance criteria there? > > On Wed, Aug 7, 2019 at 1:54 PM Elliotte Rusty Harold > wrote: >> >> >> >> On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia >> wrote: >>> >>> >>> Are these tests sufficient to say that we’re java 11 compatible? What other >>> aspects do we need to test to be able to say that? >>> >>> >> >> Are any packages split across multiple jar files, including packages beam >> dependns on? That's the one that's bitten some other projects, including >> google-cloud-java and gRPC. If so, beam is not going to work with the module >> system. >> >> Work is ongoing to fix splitn packages in both gRPC and google-cloud-java, >> but we're not very far down that path and I think it's going to be an API >> breaking change. >> > Romain pointed this out earlier and I fixed the last case of packages being > split across multiple jars within Apache Beam but as you point out our > transitive dependencies are not ready. >> >> >> -- >> Elliotte Rusty Harold >> elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: Java 11 compatibility question
On Wed, Aug 7, 2019 at 9:41 AM Michał Walenia wrote: > > Are these tests sufficient to say that we’re java 11 compatible? What > other aspects do we need to test to be able to say that? > > Are any packages split across multiple jar files, including packages beam dependns on? That's the one that's bitten some other projects, including google-cloud-java and gRPC. If so, beam is not going to work with the module system. Work is ongoing to fix splitn packages in both gRPC and google-cloud-java, but we're not very far down that path and I think it's going to be an API breaking change. -- Elliotte Rusty Harold elh...@ibiblio.org
Re: How to run DynamoDBIOTest?
FYI, I killed the job this morning after letting it run over the weeknd. Over 92 hours and no end in sight: <===--> 84% EXECUTING [92h 1m 15s] > IDLE > IDLE > IDLE > IDLE > IDLE > :sdks:java:io:amazon-web-services2:test > 4 tests completed > :sdks:java:io:amazon-web-services:test > 48 tests completed > IDLE > IDLE > IDLE > IDLE > :sdks:java:io:amazon-web-services:test > Executing test > org.apache.beam.sdk.io.aws.dynamodb.DynamoDBIOTest > IDLE > IDLE > IDLE > IDLE > IDLE > IDLE > :sdks:java:io:amazon-web-services2:test > Executing test > org.apache.beam.sdk.io.aws2.dynamodb.DynamoDBIOTest ^Celharo@elharo:~/beam$ elharo@elharo:~/beam$ On Sat, Jul 20, 2019 at 10:56 AM Elliotte Rusty Harold wrote: > > Never seems to time out or finish for me: > > $ ./gradlew :sdks:java:io:amazon-web-services:test > Starting a Gradle Daemon, 1 busy Daemon could not be reused, use > --status for details > Configuration on demand is an incubating feature. > <=<-> 98% EXECUTING [22m 21s] > > IDLE > > IDLE > > IDLE > > IDLE > > :sdks:java:io:amazon-web-services:test > 48 tests completed > > IDLE > > :sdks:java:io: > > -- > Elliotte Rusty Harold > elh...@ibiblio.org -- Elliotte Rusty Harold elh...@ibiblio.org
Re: How to run DynamoDBIOTest?
Never seems to time out or finish for me: $ ./gradlew :sdks:java:io:amazon-web-services:test Starting a Gradle Daemon, 1 busy Daemon could not be reused, use --status for details Configuration on demand is an incubating feature. <=<-> 98% EXECUTING [22m 21s] > IDLE > IDLE > IDLE > IDLE > :sdks:java:io:amazon-web-services:test > 48 tests completed > IDLE > :sdks:java:io: -- Elliotte Rusty Harold elh...@ibiblio.org
Re: How to run DynamoDBIOTest?
I've seen this myself trying to run tests in Linux (Goobuntu). On Fri, Jul 19, 2019 at 1:15 PM Anton Kedin wrote: > > Hi dev@, > > Does anyone know if there's anything extra needed to run `DynamoDBIOTest`? If > I do `./graldew :sdks:java:io:amazon-web-services:build --debug` it passes > few tests during `:test` but then seems to sit on `DynamoDBIOTest` forever. > No errors, last meaningful log is `INFO: Container > localstack/localstack:0.8.6 started`. Happens on different machines, both on > master and release-2.14.0 branches. > > Any pointers? > > Regards, > Anton -- Elliotte Rusty Harold elh...@ibiblio.org
